Developing eWrite

If anyone wants to now how Not to send bulk emails, come talk to me. I recently had difficulty getting a particular clients email newsletter into their subscribers email inboxes, this costed our business a customer.

While working on the clients newsletter I decided to start from scratch to do everything I could to ensure the software and server were as valid and as transparent as possible to ensure no emails were being labelled as spam. This kicked my ass for a few days and I had some frustrations along the way. Bizarre bugs cropped up with every single test email sent. Sometimes the content displayed the HTML rather than rendering it, other times the ‘to’ name and ‘from’ name didn’t bother displaying.

Rather than more ranting about the problems, Here is a list in no particular order of the actions I took in the hopes that they will help some other company get their emails into inboxes or improve their own software.

Hosting

I Set up a new Virtual Private Server (VPS) with Blacknight - http://www.blacknightvps.com/. This provided me with a static IP address and allowed me to set up reverse DNS.

SPF Record

I Set up an SPF record for our domain name using http://www.openspf.org/.

White Lists

Requested to get our domain name listed on abuse.net

Reporting Tools

Signed up to and paid dnsstuff.com to access to their DNS testing tools. I was able to solve all problems that the reports pointed out.

Signed up to Microsoft’s Smart Network Data Services to receive data about the traffic seen originating from my IP address, such as mail volume and complaint rates.

Email headers

Added new (to me) headers to the out going emails.

Precedence: bulk
List-Id: xxxxx <yyyyy>
List-Owner: <mailto:owner@example.com>
List-Subscribe: <http://www.example.com/path/to/subscribe.php>
List-Unsubscribe: <http://www.example.com/path/to/unsubscribe.php>
List-Help: <http://www.example.com/path/to/help>

Thank you to the staff of blacknight for recommending these headers.

Existing Headers

Date: $date
To: Recipient-Name <Recipient-Email-Address>
From: Recipient-Name <Recipient-Email-Address>
Reply-To: Recipient-Name <Recipient-Email-Address>
Return-Path: Recipient-Name <Recipient-Email-Address>
Message-ID: <".time()."-Recipient-Email-Address>
X-Mailer: PHP v".phpversion()."
MIME-Version: 1.0

The position of ‘MIME-Version: 1.0′ is important. There was an excellent page about this on a site but I can’t find it now unfortunately. Some emails may not display correctly in Microsoft products such as Outlook if this header is the first header.

Reading

I read many forum posts with others also experiencing similar email marketing problems.

Referred to the SpamAssassin tests when encountering any failed tests http://spamassassin.apache.org/tests_3_2_x.html

A older blog post from Michele Neylon regarding the Do’s and Don’ts of mass emailing.

Spam assassin - tips for legitimate senders.

Spamhaus Frequently Asked Questions.

Testing

Sent test after test after test after test to myself and several other email addresses on various domains to test being labelled as false positives and layout issues.

When sending a HTML email, I was getting the PHP code to create a nice and neat text-only version. This would strip the HTML tags and then strip out all the spaces left behind. In my mind I was creating a nice and neat text version. I learned however that leaving the spaces there after trimming the HTML is best, SpamAssassin thinks the content is too different to the HTML if its trimmed back.

Sent test emails to auth-results@verifier.port25.com which provide a free automatic response. In their own words : “The service allows email senders to perform a simple check of various sender authentication mechanisms. It is provided free of charge, in the hope that it is useful to the email community.”

Wrote several custom functions to alter the layout of the emails slightly to allow the email to be displayed properly in some email domains’ web based email interfaces such as the eircom webmail interface.

When defining email headers, use ‘\n’ for line breaks on *nix servers and use ‘\r\n’ on Microsoft servers. When using the incorrect type the email headers may not display at all for some recipients.

Updated our software to send large numbers of emails in smaller chucks so as not to overwhelm anything.

Also added some helpful alerts to let a user know if they are using words or phrases that SpamAssassin doesn’t like and may lead to their email being labelled as spam.

I had problems with my ISP too, the DNS wasn’t refreshing after several days to I began using OpenDNS.org DNS details. Sadly I only found out about this close to the end of my problems but they certainly helped. At one point I couldn’t FTP into the site to make any changes at all.

Result

Our software now sends out a mean email. Unfortunately too late to convince a lost customer. Confidence is low at the moment but I will be working to continue improving the software in any way I can. I will be stricter in what the software allows a user to send out. If anyone has anything else that would help in sending emails or reporting errors, please let me know.

Related articles by Zemanta

• Register365 Support Fail
• How-To: 5 Ways to Increase Email Deliverability
• PHP: Sending Email (Text/HTML/Attachments)

The Tuesday Push is an initiative started by Damien Mulley to encourage the blogging community to point a spotlight on new or useful services in Ireland.

The Tuesday Push happens every second Tuesday. This particular Tuesday isn’t one of them but I wanted to promote on of my own projects and how it relates to another of Damien’s ideas, the SEO competition.

In short, the SEO competition aims to find out who is the best at search engine optimisation in Ireland. I think the competition is an excellent idea and several sites and blogs are already appearing to compete.

Im interested in the tools and techniques of SEO but I’m not participating myself, I don’t know enough. Im going to be watching the participants and their sites so that I might learn a thing or two to benefit our clients. Hopefully, when this competition ends in December, one or more people will blog about the techniques they used so we might all learn from it.

My own project that I wanted to talk about is ‘Yell if it changes‘. This site was originally started so that I could be notified automatically if some of our customers sites had changed. For a long time I planned to add the ability to monitor the Google Search Engine Results Page (SERP) for changes in the results for specific keywords and URLs, This is now in place.

For the SEO competition, enter the magic words ‘Geansai Gorm‘ and the URL you would like to monitor. You will be notified of the current position(s) in the results page of Google and you will also be notified via email whenever the URL changes position(s). The history of changes in positions are recorded too so it should make for some very interesting data in time.

It isn’t limited to the SEO competition though, anyone can use it to monitor the positions of their own sites for their chosen keywords. Go on, its free and you can use it for as many sites or search terms as you like.

Yell if it changes is a small project I work on in my spare time. There are many many things I plan to add to it in the future to make it more useful to myself and the 200+ users so far. If you have any ideas or comments on Yell, I’d love to hear them.

I’ve recently read a few blog posts talking about how Google will soon be offering RSS feeds for basic search results.  I thought this was very cool. As I began thinking of the various uses for this I decided I couldn’t wait for Google to bring it out so I created a basic facility using PHP to provide Google search results in RSS form.

Try it out:

Google.ie search results in RSS form

Check it out, see what you think. At the moment I’ve limited it to 50 results. In the coming days I’ll add to it, get the feeds to automatically update every 24 hours or so. I’d like to add some sort of indicators to show if a website URL has moved up or down and to keep track of some of my own URLs. If anyone wants the PHP code for this, let me know.

Some related articles :

• Finally! Google to Offer RSS Feeds for Web Search Results
• Google Confirms RSS For Web Search Results
• Google adding RSS feeds for web search results: mixed emotions

Related articles by Zemanta

• Google is introducing RSS enabled search real soon
• Google Web Search May Get RSS Feeds via Google Alerts

This is a small but effective addition to our eWrite products added this morning.

We often create small applications for our customers, items such as order forms or any other interactivity a user wants to put on their website. These items need an interface to allow the owner to manage them.

Ewrite can now detect any applications we have created and puts a new menu button into the top of the main interface. This new menu item called ‘Apps’ shows any applications we have created for the user and automatically links to any relevant sections of the application.

This allows an eWrite user to access any apps we create for them from within eWrite so that they don’t have to note or remember several URLs, usernames or passwords.

This also allows the applications we create to inherit the style, security and resources already available within eWrite which reduces the amount of code and files needed in these applications.

The applications are also stored in such a way that when eWrite is updated from SpringLoops and it won’t negatively effect the installed applications.

As a developer Im quite aware that I should have begun to use version control a long long time ago to develop our apps.

More than once I read about and tried to use CVS or SVN. The learning curve to set these systems up on a server is probably not too steep but I was continually dissatisfied and turned away from it.

I wrote my own version control system at one point using PHP to monitor changes in the files and collect any comments in the changed files. The PHP code could also upload the changed files to the users servers. This worked quite well for a time but I was worried about security and I had a tendency to get bogged down in the development of the system itself instead of what it was supposed to do to help me in my daily work.

Earliy in September I came across SpringLoops and found they offer a free version of their source code management tools. Their free version allowed me to create 1 repository so I signed up and created a repository for eWrite Lite.

I installed TortoiseSVN on my desktop machine and on my laptop and got used to downloading and updating the code from both locations and getting familiar with the terminology.

I’ve read plenty of tutorials and blogs where developers encourage the use of version control. More than once a developer would write “Use version control, you can thank me later”. The smug feckers were right. I’ve been using SpringLoops and TortoiseSVN for a few weeks and its allowed me a freedom to concentrate on the development of the software rather than fiddling around with my own bad habits.

I used to have HP servers in the attic running Ubuntu Server with Apache, PHP and mySQL to host, develop and backup everything I was developing. I sold off all the servers, replaced a dodgy router and ran CAT5e 100MBPs cable through the building rather than using a wireless connection.

On my desktop machine I use WampServer to deploy a development environment of Apache, PHP and mySQL in Windows XP. I can write and test the code on my local machine and use TortoiseSVN to commit the changes to SpringLoops and deploy the stable versions to customers sites.

Im still learning my way around SpringLoops and version control abilities in general. I’ve upgraded to the ‘Field’ package on SpringLoops to allow for more repositories and servers to deploy to.

The packages that are available are more than enough for my needs in some cases but lack a little in others. For example the ‘Field’ package Im using now has 10GB of space, 100 protects and 20 servers per project. The space and projects are way more than I need, but the servers per project are important to me, I need more.

The next package up allows for only 30 servers. I emailed SpringLoops to ask if there were customized packages available. They seemed helpful at first asking what I would need. I asked for at least 50 servers per project and far less space and projects. I’d be more than happy with 10-15 projects and 2-5 GB of space. I didn’t get any reply to that, they probably think Im joking. I’ll contact them again shortly to see if they’ll allow it.

Developing with SpringLoops for the past few weeks has been great. I’ve been able to develop the eWrite products much faster, commit changes to the server and deploy these changes to clients sites with just a couple of clicks, It is far far faster than FTP. I wish I had joined the version control party sooner, the eWrite software and my time are all the better for it.

When we meet new customers for the first time they often have existing websites that they would like to improve upon. In this case, we can import their existing website pages into a database and install eWrite behind their existing website where they can log in and edit their pages when ever they like.

However, up til now, for new customers with no existing websites we have had no default website structure to provide a user to begin creating their website. We have found it is useful for a user to have a plain website to start from. The idea is that a user would use this default structure and become familiar with using eWrite while we or some other designer works with them to create a professional looking website.

The image below shows the new default web page template that a user begins with when they have set up a brand new website with eWrite Lite. Almost everything on that page can be edited by the user. When installing eWrite Lite, the user is asked a few questions such as business name, website address, website description, business keywords, website owner and so on. This information is used to install eWrite Lite and create the default website.

eWrite Lite default template

The user can then log into eWrite Lite and begin creating new pages. Links to these new pages are automatically added to navigation panel on the left. Other items on that page can easily be turned on or off by the user. For example if the user would prefer not to show the page created / modified date they can simply disable them.

Behind the scenes, eWrite lite also creates an XML sitemap for search engines like Google to index your site. A RSS feed is also created for those who would like to keep up with changes to a site using their favourite RSS reader.

Search engine optimisation is also catered for by ensuring users fill in proper page names, page descriptions and keywords when creating or editing pages.

The aim is to provide a new user with a solid base to start their new site. I would love some feedback from web developers on this default template approach. Do you use something similar?

A new forum has been set up at http://www.ewritecork.com/forum/.This forum is to encourage ewrite users and members of the public to discuss or comment on anything to do with eWrite.

If you have any questions or comments about ewrite, nows your chance! Get posting, any and all suggestions are welcome.

In the very near future, the ewrite software will be available to buy online, so this forum will be especially useful to people out there downloading and using ewrite.

The forum itself comes from PHPBB2. Its a particular favourite of mine used by many sites out there. They have a more up to date forum, version 3 which has more facilities and is a little sleeker looking, I prefer version 2 though becasue I know my way around it.

A few weeks ago, a web service called Toddle was the focus of a Tuesday Push. This gave me the opportunity to sign up to Toddle and see what it could do.

In a nutshell, Toddle allows you to create great looking email newsletters very quickly. I saw that Toddle newsletters could be imported into Campaign Monitor and I wondered if eWrite Messenger could do the same.

With Toddles permission we have created an import feature in eWrite Messenger which allows you to import the newsletters you have created into eWrite Messenger. From here, you can send out the newsletter to anyone in your address book and monitor the readers.

Im very happy to be able to work with another Irish company, to be able to bring another great Irish product to eWrite users. Thanks Toddle!

Related articles by Zemanta

• Toddle - Make Your Email Newsletter Stand Out
• Tuesday Push : Toddle.com From Alan O’Rourke
• Growing a Mailing List for Free (well, Almost)

A good friend and fellow developer Matt Hart has created a handy tool for helping developers find unclosed HTML tags on their sites.

HTML Tag Checker can check the HTML tags of online web pages. It will tell you how many errors there are, and highlight them within the structure of the HTML.

To date it has checked a total of 141 web pages and found 1923 errors.

Matt recently gave me a hand to get the eWrite Lite Demo online to allow users to try out the content manager using some of their own pages.

Matt is one of those frustratingly intelligent guys that if left alone for 5 minutes with a computer and an idea will create something great. Recently he has also created a couple of fun games, Battleships-Online and Boondog.

Inspired by Matts interest in Parkour (free running) Boondog has received some great attention in a short time. Written in FreeBasic, it has been downloaded nearly 700 times and was recently reviewed on bytejacker.com.

Battleships-Online is my favourite though, in between jobs I often nip off to the site and take a couple of turns when I should be working.

Looking forward the next app out of Matt..

eWrite Messenger Login

Recently I have been working on ‘eWrite Messenger’. It allows a business to build up and manage an address book of email addresses and send out email newsletters to as many people as they like.

The main feature of eWrite Messenger is being able to send out these emails and record if readers are receiving the emails, if they are reading the emails and what links are they clicking.

One of the methods used to record this information is using PHP to add a unique email ID and a user ID into each email. When the email is loaded a PHP script disguised as an image is loaded and the email ID and user ID are passed to it via a querystring. The script then saves this information which allows the progamme to know which user opened which email.

eWrite Messenger - Recently sent emails

This process is well known and often blocked by many anti virus programmes either on the users email server or the server sending out the emails in the first place.

My question is WHY???

As always there is probably a way to abuse this approach to rob a persons emails or address book. Why isn’t there an accepted form of logging information from an email? We all have Google Analytics or similar on our sites and blogs which log far more information about every visitor. Why isn’t this allowed in email?

Adding this information to links in an email still works. This allows a programme to see what link was clicked by a user. It doesn’t allow a programme to log if an email was opened however and so any user who opens an email but doesn’t click on the links isn’t logged.

Another method I want to try out is using server logs. For every email newsletter created and for every user in the address book a unique image is created, probably something very small and insignificant. When a newsletter is sent out it embeds the approprate image into the email. Not a PHP script disguised as an image, just a regular image. Then using the server logs one can see what images were requested from the server.

The problem with this is that the servers don’t allow PHP code to access the logs. 2 hosting companies I use both have logs that a user can access by FTP but don’t allow PHP code to access them. Blacknight only keeps logs for 5 days!

To me, allowing PHP to access the logs to determine if specific images were loaded allows for a great way to work out what emails were loaded and by which users. No progamme or user has to worry about anti virus programmes getting in the way since they are normal everyday images being loaded.

Anyone else work in this area and have any comments or suggestions?