Developing eWrite

If anyone wants to now how Not to send bulk emails, come talk to me. I recently had difficulty getting a particular clients email newsletter into their subscribers email inboxes, this costed our business a customer.

While working on the clients newsletter I decided to start from scratch to do everything I could to ensure the software and server were as valid and as transparent as possible to ensure no emails were being labelled as spam. This kicked my ass for a few days and I had some frustrations along the way. Bizarre bugs cropped up with every single test email sent. Sometimes the content displayed the HTML rather than rendering it, other times the ‘to’ name and ‘from’ name didn’t bother displaying.

Rather than more ranting about the problems, Here is a list in no particular order of the actions I took in the hopes that they will help some other company get their emails into inboxes or improve their own software.

Hosting

I Set up a new Virtual Private Server (VPS) with Blacknight - http://www.blacknightvps.com/. This provided me with a static IP address and allowed me to set up reverse DNS.

SPF Record

I Set up an SPF record for our domain name using http://www.openspf.org/.

White Lists

Requested to get our domain name listed on abuse.net

Reporting Tools

Signed up to and paid dnsstuff.com to access to their DNS testing tools. I was able to solve all problems that the reports pointed out.

Signed up to Microsoft’s Smart Network Data Services to receive data about the traffic seen originating from my IP address, such as mail volume and complaint rates.

Email headers

Added new (to me) headers to the out going emails.

Precedence: bulk
List-Id: xxxxx <yyyyy>
List-Owner: <mailto:owner@example.com>
List-Subscribe: <http://www.example.com/path/to/subscribe.php>
List-Unsubscribe: <http://www.example.com/path/to/unsubscribe.php>
List-Help: <http://www.example.com/path/to/help>

Thank you to the staff of blacknight for recommending these headers.

Existing Headers

Date: $date
To: Recipient-Name <Recipient-Email-Address>
From: Recipient-Name <Recipient-Email-Address>
Reply-To: Recipient-Name <Recipient-Email-Address>
Return-Path: Recipient-Name <Recipient-Email-Address>
Message-ID: <".time()."-Recipient-Email-Address>
X-Mailer: PHP v".phpversion()."
MIME-Version: 1.0

The position of ‘MIME-Version: 1.0′ is important. There was an excellent page about this on a site but I can’t find it now unfortunately. Some emails may not display correctly in Microsoft products such as Outlook if this header is the first header.

Reading

I read many forum posts with others also experiencing similar email marketing problems.

Referred to the SpamAssassin tests when encountering any failed tests http://spamassassin.apache.org/tests_3_2_x.html

A older blog post from Michele Neylon regarding the Do’s and Don’ts of mass emailing.

Spam assassin - tips for legitimate senders.

Spamhaus Frequently Asked Questions.

Testing

Sent test after test after test after test to myself and several other email addresses on various domains to test being labelled as false positives and layout issues.

When sending a HTML email, I was getting the PHP code to create a nice and neat text-only version. This would strip the HTML tags and then strip out all the spaces left behind. In my mind I was creating a nice and neat text version. I learned however that leaving the spaces there after trimming the HTML is best, SpamAssassin thinks the content is too different to the HTML if its trimmed back.

Sent test emails to auth-results@verifier.port25.com which provide a free automatic response. In their own words : “The service allows email senders to perform a simple check of various sender authentication mechanisms. It is provided free of charge, in the hope that it is useful to the email community.”

Wrote several custom functions to alter the layout of the emails slightly to allow the email to be displayed properly in some email domains’ web based email interfaces such as the eircom webmail interface.

When defining email headers, use ‘\n’ for line breaks on *nix servers and use ‘\r\n’ on Microsoft servers. When using the incorrect type the email headers may not display at all for some recipients.

Updated our software to send large numbers of emails in smaller chucks so as not to overwhelm anything.

Also added some helpful alerts to let a user know if they are using words or phrases that SpamAssassin doesn’t like and may lead to their email being labelled as spam.

I had problems with my ISP too, the DNS wasn’t refreshing after several days to I began using OpenDNS.org DNS details. Sadly I only found out about this close to the end of my problems but they certainly helped. At one point I couldn’t FTP into the site to make any changes at all.

Result

Our software now sends out a mean email. Unfortunately too late to convince a lost customer. Confidence is low at the moment but I will be working to continue improving the software in any way I can. I will be stricter in what the software allows a user to send out. If anyone has anything else that would help in sending emails or reporting errors, please let me know.

Related articles by Zemanta

• Register365 Support Fail
• How-To: 5 Ways to Increase Email Deliverability
• PHP: Sending Email (Text/HTML/Attachments)

The past few days have been particularly tough. I’ve been working on a new clients email newsletter making sure they can get their monthly newsletter into their recipients inboxes.

The clients site is hosted with Register365.  One of the measures to make sure the email gets through is to add an SPF record. To do this I need to update the DNS settings on the clients site. I wanted to search Register365 support to see if I could do this.

Just when I need it most, the support section of Register365 was offline and still is.

Register365's "Support"

Using Google Cache, I can see what it should look like and that it was at least online on November 1^st when Google cached it. Luckily I have Yell if it changes, which is going to send me an email when the site is back online.

Register365 Support on Google's Cache

I ranted about this on Twitter on Friday evening and I was very impressed when I received a message from Stephen McCarron, managing director of Hosting365 asking if he could be of any assistance. Unfortunately Stephen couldn’t help, but I was reminded yet again how Twitter can be a valuable business resource. Thanks for contacting me Stephen.

Blacknight have been the complete opposite to Register365. I signed up to a VPS account allowing me to have a static IP address, set up reverse DNS and more. I talked to several members of Blacknight at different times using their Live Chat and email and every time I was answered promptly and came away a happy customer.

To further prevent me from doing any productive work the last few days, my ISP has been terribly slow to update their DNS records. I pointed a domain name to the new VPS account. Within hours the site was running for anyone I asked to look but when looking at the domain name from my office it still hadn’t updated for 4 days despite flushing my local DNS and resetting the router a couple of times.

So for a frustrating few days when I really needed things to work my own ISP and Register365 failed me. If it wasn’t for Blacknight, their services and staff I’d be completely fecked and I would have most likely lost a valued customer. Thank you Blacknight.

eWrite Messenger Login

Recently I have been working on ‘eWrite Messenger’. It allows a business to build up and manage an address book of email addresses and send out email newsletters to as many people as they like.

The main feature of eWrite Messenger is being able to send out these emails and record if readers are receiving the emails, if they are reading the emails and what links are they clicking.

One of the methods used to record this information is using PHP to add a unique email ID and a user ID into each email. When the email is loaded a PHP script disguised as an image is loaded and the email ID and user ID are passed to it via a querystring. The script then saves this information which allows the progamme to know which user opened which email.

eWrite Messenger - Recently sent emails

This process is well known and often blocked by many anti virus programmes either on the users email server or the server sending out the emails in the first place.

My question is WHY???

As always there is probably a way to abuse this approach to rob a persons emails or address book. Why isn’t there an accepted form of logging information from an email? We all have Google Analytics or similar on our sites and blogs which log far more information about every visitor. Why isn’t this allowed in email?

Adding this information to links in an email still works. This allows a programme to see what link was clicked by a user. It doesn’t allow a programme to log if an email was opened however and so any user who opens an email but doesn’t click on the links isn’t logged.

Another method I want to try out is using server logs. For every email newsletter created and for every user in the address book a unique image is created, probably something very small and insignificant. When a newsletter is sent out it embeds the approprate image into the email. Not a PHP script disguised as an image, just a regular image. Then using the server logs one can see what images were requested from the server.

The problem with this is that the servers don’t allow PHP code to access the logs. 2 hosting companies I use both have logs that a user can access by FTP but don’t allow PHP code to access them. Blacknight only keeps logs for 5 days!

To me, allowing PHP to access the logs to determine if specific images were loaded allows for a great way to work out what emails were loaded and by which users. No progamme or user has to worry about anti virus programmes getting in the way since they are normal everyday images being loaded.

Anyone else work in this area and have any comments or suggestions?

I placed an ad on facebook for 10 days to run from July 15^th to July25th. I specified that I would pay 1 Dollar for a Click and that the maximum I would pay per day would be 10 Dollars.

There were a few differences in the number of clicks when I compared Facebook ad details to Google analytics so I went investigating.

Unfortunately, I don’t have access to the logs from Blacknight to allow for a third source of data, logs are kept only for 5 days. If I advertise again on Facebook, Ill include my own scripts to log traffic data too.

A chart of the Ad clicks according to Facebook ad manager and Google Analytics.

According to Facebook Ad manager, the ad was clicked on 66 times. The ad ended on the 25^th but it tells me the ad was still clicked on on the 26^th which is nice.

The numbers in red show the days where Facebook ad manager and Google Analytics disagree about the number of clicks and visits. 8 of the overall 13 days for a 10 day ad campaign show different numbers

On the 15^th, my site was changing hosting over to Blacknight so that probably explains the difference of 3 visits that day. The site may not have loaded correctly for the user that day and so Google probably didn’t record the visits properly.

On the 26^th and 27^th, I don’t know why Facebook still had the ad going on the 26^th, I got charged for it which went over the free 100 dollars I gots from the VISA application and my credit card was charged accordingly.

Facebook charge of €13.21 on AIB 24 Hour banking

According to Google, I received 2 referrals from Facebook on 26^th and 27^th. Based on a comment from John O’Connor that never occurred to me, most likely the 2 referrals on the 27^th is a result of 4 visitors on the 26^th and they didn’t navigate away from the site or close the site until after midnight. Its a pity I cant see specific times on Google. Perhaps the exported reports from Facebook ad manager will tell me, but I can’t open it properly at the moment since the .xls file messes up when I open it in Open Office.

For the rest of the days, I can’t explain the differences between Facebook ad clicks and Google analytics visits.

In total, I was charged 120 Dollars for the ad campaign. I assumed (incorrectly?) that I would only be paying if the ad was clicked so according to the clicks shown above I should only have to pay 66 Dollars as it was 1 dollar per click.

I told the Ad manager I would like to spend no more than 10 Dollars per day so it seems I was charged the full 10 Dollars per day regardless of the number of Clicks.

Has anyone else advertised on Facebook? If you’re considering it, read Damien Mulleys tutorial on Advertising on Facebook and read Alexia Golez’s post about getting money for free to advertise.

I recently wrote about moving to greener pastures, virtually speaking. This blog is now hosted with Blacknight.

The reason for the move is that my previous hosting was on Microsoft Windows servers and simply put, it didn’t work.

Any Wordpress version above version 2.3.1 resulted in plenty of errors, blank screens upon logging in, not being able to use pretty permalinks, categories not loading etc. And not just on my blog.

The site is now hosted on a linux server and its running beautifully. I’ve been able to upgrade Wordpress to the latest version without problems and I’ve enabled pretty permalinks.

I changed over the hosting on Tuesday, which I stupidly timed with creating my first advert on Facebook for eWrite Lite. The ad created plenty of clicks and plenty of them might have received a blank site thanks to my own fantastic timing as the site changed over to BN DNS name servers. But thats for a later blog post

For now, Thank you Blacknight! I’m recommending customers using eWrite Lite CMS to use Blacknight hosting.

Im about to move this blog and site over to Blacknight hosting. I’ve paid for a small package and I’ll be changing the name servers shortly.

I’ve had some trouble recently setting up blogs on the existing hosting which is using IIS and so I’m looking forward to residing on a Linux server which will enable me to upgrade the blog to version 2.5.1 and enable the friendlier perma links which seem to cause issues for IIS.

Once there, I’ll be continuing to blog about the ongoing developments of eWrite Lite, eWrite Messenger and eWrite Forms more frequently.Who knows, I might end up being featured on Alltop!

Tomorrow morning I will be emailing a potential client who wants to use eWrite for their website to tell them I can’t help them.

Their site is using ASP.NET and Microsoft SQL Server. I currently only have eWrite Lite using PHP and mySQL.

So my options are to let them know that I don’t have a version of eWrite that will work on their site or else try to get them to host their site with Blacknight.

I’ve contacted their service provider in England. They don’t have mySQL or PHP on their servers and don’t plan to any time in the future. Bummer.

Im fairly sure they won’t want to change their site hosting as they have a close relationship with their current providers and also have other website related databases hosted there.

To add insult to injury, when I demoed eWrite Lite to them they expressed an interest in franchising eWrite Lite in relation to a web design company they are involved in. This would be great for eWrite Lite but alas I don’t think they’ll be jumping up and down about it right now.

As a result of this I’m wondering should I build a .NET version of eWrite Lite? I worked with ASP long before I worked with PHP and so getting into .NET wouldn’t be beyond me. I’m out of the loop on MS stuff though so wondering what costs I will be facing if I go down this road. Do I need a licence of any kind to write .NET apps? Can anyone recommend a good IDE for .NET ? I wonder is there an app out there to convert PHP code to .NET code which might save some time