VPS or Dedicated - Fedora Core or CentOS 4? Control panel?

[BGSWebDesign]

Hi,

I'm finally moving to a new server, I've got a few choices to make, can anyone help me sort this out.

First a little bit on what I'm running now: I run about 150,000 in my lists (all total), but when I do a mass email to everyone and remove dupes I'm down to about 55,000 - just so you know, this all runs fine on a shared server, that is until this afternoon.  My host has moved everyone to a new control panel - JOY, JOY, that new panel does not allow setup of procmailrc to script emails, so basically I'm Screwed!  what does the hosting company care - I don't know I've been with them over 3 years, I guess they don't... so now a few choices left for a Monday Morning (I can tell this is going to be one hell of a fun week).

1.) Should I choose a VPS server ($20/month), or a dedicated server ($60-$80/month)?   Here's a little detail on each:

VPS Memory 256MB guaranteed, 15GB disk space, 200GB bandwidth.
Dedicated 512MB, 80GB disk space, 900GB bandwidth.

2.) IF I choose dedicated which OS is best?
  Fedora Core or CentOS 4  (these are my only 2 choices)

3.) Do I need a control panel, should I even care about one, or not?

That's it, thanks for the input - and may all of your hosts never decide to move you to a Control Panel of their liking 'on a whim'...

[AlanT]

Would it be possible to have the server administrator set up the procmailrc to script emails?  Then you wouldn't have to move.  And since you're asking about whether you even need a control panel, if you know how to manage the server via SSH, then you should be set no matter where you go.

I use a VPS with Fedora Core, but with a much smaller list.  From my experience, it's more important who you host with than with what software.  After moving through various hosting companies over the years, I'm very happy with ServInt.com.

CPanel has been a very easy control panel for me to use.  I haven't used many, but this one I like.

[DW]

Brett,

I recommend talking to your existing host about avoiding the move, if possible.  As Alan mentioned the host may very well be able to assist with setting up the required email forwarder manually.

Whether a VPS is worth it really depends on how much email other users on a VPS send, which is practically impossible to predict.  In my experience if you are sending from a single IP and/or domain it is not good to send more than about 100,000 - 150,000 emails per day.

I prefer Fedora Core but CentOS is similar and, I'm sure, just as reliable.

If you are not familiar with command-line administration of a server then yes, I highly recommend a control panel.

Regards

[BGSWebDesign]

Hi,

I recommend talking to your existing host about avoiding the move, if possible.

Unfortunately this was NOT an option, they moved everyone, there was no choice.  

Now, I've got other problems, but they seem to be from before...  my delivery rates/click-throughs are very low, and I have to find a way to simulate a procmail script forward (going to try that with Perl).

For example I've been hitting 55,000 emails on my most recent list, and I'm lucky to get 500 clicks, is this normal, below normal, or above normal?

I'm not positive, but it seems on the very low side, the clicks have been slowly dissolving away - I'll be lucky if I can stay in business another month or so with this type of response, secondly the followup clicks that lead to purchase are also dissolving...

I've purchased a dedicated server (*that I have NOT yet setup) to see if it makes any difference, they set me up on CentOS I believe because they said the Fedora issues updates every few months.

Will Dedicated give me a better delivery/click-through rate?  I hope so as I'm about to go out of business....  I've heard of lists degrading over time and becoming unresponsive - but this is NUTS...

Anyone have any thoughts/comments/ideas???

[DW]

Brett,

that new panel does not allow setup of procmailrc to script emails

I highly recommend asking your host if they can manually set up the required forwarder to bounce.cgi.

I've been hitting 55,000 emails on my most recent list, and I'm lucky to get 500 clicks, is this normal, below normal, or above normal?

It's impossible to say if this is normal  or not... To be sure of delivery your host could run a log analysis tool.  www.sawmill.net has a free trial that can be used - it allows you to see daily results for a chosen "From" address, such as your ListMail bounce address.

Apart from that it's possible that the reputation of URLs used in your messages is causing temporary or permanent blocks.  The same can happen with your IP, with the likelihood of blocking increasing with the number of users sending email from a shared server.

A dedicated server will allow you to run sawmill yourself to see true final delivery rates...

Regards

[BGSWebDesign]

DW,

Thanks for the input, a few questions...

I highly recommend asking your host if they can manually set up the required forwarder to bounce.cgi.

Actually, no I handle bounces through a bounced email account as I found that to be a better way to do things...  still, I'm in the process of setting up the Dedicated Server and I want to see the performance difference, ie, I want to remove the 'throttling' code if I can and see if I get more clicks.

Regarding SawMill you said:

To be sure of delivery your host could run a log analysis tool. www.sawmill.net has a free trial that can be used - it allows you to see daily results for a chosen "From" address, such as your ListMail bounce address.

I didn't know there was anything that could show me 'delivery', are you saying that SawMill will let me see how many people actually 'receive' the email being sent - and can you briefly tell me how to configure/set that up?   I've just downloaded the Red Hat version of SawMill and will be installing it.

Apart from that it's possible that the reputation of URLs used in your messages is causing temporary or permanent blocks. The same can happen with your IP, with the likelihood of blocking increasing with the number of users sending email from a shared server.

Here's the funny thing, ALL of the URLs in the message point to the domain of where I am sending email from using LMP - I use the Link Tracking feature of LMP - I do this because I want the links in my emails to be consistent with the domain of where they are being sent from, so I don't think this would have any effect?    

Secondly, this was happening way before they moved me to this new server - but it does seem to be worse now that I'm on the new server.

Regarding blocking increasing, as far as I know I'm not listed on any blocked sites, I have a subscription to a service that checks that, just a min and I'll run it...  ok, I'm not on any blacklists, here are the ones checked:

Code: [Select]

MAPS RBL
MAPS DUL
MAPS RSS
ORDB
ORBZ
ORBL
dorkslayers
/dev/null
Osirusoft
Local Black Holes (50 of them)
uceb.org SPAM blacklist
blackholes.us
JIPPG's Relay Blackhole List Project
The Abusive Hosts Blocking List
NJABL.ORG
SORBS
no-more-funn
RRBL
Spamhaus SBL
Leadmon.Net's SpamGuard Listings
Spamhaus XBL
SpamCop

Here is my result:

Code: [Select]

The IP address (xx.xxx.xxx.xxx) was NOT FOUND in any blacklists searched.

Are there other sites/blocking you are referring to - such as AOL/HotMail/Gmail, etc...  I know I have AOL Whitelisting enabled, but maybe it's not the correct IP after the move?  Any suggestions on setting that up properly?

Thanks for you help - also, as I'm moving to Dedicated, I'll probably have questions on optimizing the emailing - I'd like to get up to 20,000/hour if I can, BUT, not if it results in missed clicks - which is why I've continued to run throttling code in the 'shared environment', we'll see...  

Regarding the OS, it's CentOS which I'm told is a variant of RedHat 4.4.

[mike2]

Apart from that it's possible that the reputation of URLs used in your messages is causing temporary or permanent blocks. The same can happen with your IP, with the likelihood of blocking increasing with the number of users sending email from a shared server.

Here's the funny thing, ALL of the URLs in the message point to the domain of where I am sending email from using LMP - I use the Link Tracking feature of LMP - I do this because I want the links in my emails to be consistent with the domain of where they are being sent from, so I don't think this would have any effect?    

Are there other sites/blocking you are referring to - such as AOL/HotMail/Gmail, etc...  I know I have AOL Whitelisting enabled, but maybe it's not the correct IP after the move?  Any suggestions on setting that up properly?

The URL of your LMP is subject to the same standards as any other URL... therefore it could be causing problems.  You should check your IP and URL for black listings.

AOL whitelisting is definitely tied to the IP's so if you have moved then I can pretty much guarantee it's not in effect.  It may actually be causing worse problems if you had your domain tied to the old IP and have moved  to a new ip.

Reset it up with them for your Dedicated server, not the shared host you are on now ( unless you have a dedicated IP ).  Same for Yahoo and MSN.

On another note... the sawmill might help show initial delivery, but even that in NO WAY guarantees the person actually saw the email or actually even got it.  Many mail systems will accept the mail, which to your system seems like it was delivered.  But then they process it and run it through checks, blacklists, etc and if it fails just discard it...

[DW]

Brett,

that new panel does not allow setup of procmailrc to script emails, so basically I'm Screwed!

Since you said you aren't using bounce.cgi does this mean you have signup scripts?  (If not, why do you need to use procmailrc?)  If you use signup-by-email I still recommend contacting your host about the possibility of them manually setting up email forwarding to CGI.

I want to remove the 'throttling' code if I can and see if I get more clicks.

I think throttling can only help delivery, so if all the messages are getting to the server it should only be a few days until the majority of clicks come in.  Removing throttling IMHO should not improve things.

Are you saying that SawMill will let me see how many people actually 'receive' the email being sent

Yes.  It shows the count of "Success" and "Failures" for final delivery.  I have struggled with providing such log results with my own scripts, especially considering the wide variety of server email software (qmail,sendmail,exim,postfix,etc)

And can you briefly tell me how to configure/set that up?

1. Login as root to your server
2. Change to a work folder with enough hard drive space (check with df -h)
cd /usr/src
3. Download sawmill.  Get the "encrypted source" copy so it works on any system:
wget http://www.sawmill.net/download/sawmill/7.2.9/sawmill7.2.9_crypt_src.tar.gz
4. Unzip and compile
tar xzf sawmill7.2.9_crypt_src.tar.gz
cd sawmill-7.2.9/
./configure
make
5. Find your maillogs for your reference
locate maillog
6. Verify this is the correct maillog.  If you get output containing recent timestamps (date will give you the current time) it should be the correct file.
tail /path/to/maillog
6. Run the program.  This will lock up your SSH session.  (Press Ctrl-C to shut down sawmill.)
./sawmill
7. Browse to the given URL+port to start setting up sawmill, import your log files, etc.

As far as I know I'm not listed on any blocked sites, I have a subscription to a service that checks that

You may want to find a service that checks even more services for blocks.  This one seems pretty good - I've seen a better/similar one, I think, but can't find it now.

Are there other sites/blocking you are referring to - such as AOL/HotMail/Gmail, etc... I know I have AOL Whitelisting enabled, but maybe it's not the correct IP after the move? Any suggestions on setting that up properly?

In my experience AOL will not whitelist you unless you are the sole-sender from your server.  I doubt that shared hosts will have been whitelisted... I have not seen widespread delivery problems with AOL.   I have seen some delays on servers sending upwards of 100k/day but very rarely a full block.  They are a lot more relaxed than Yahoo, who I have noticed is doing some very strict rate-limiting these days.  Without special permission I don't think you can deliver more than about 25,000 messages a day per server to Yahoo (this is a very rough guess - I have never timed the acceptance of emails...)

I'd like to get up to 20,000/hour if I can, BUT, not if it results in missed clicks - which is why I've continued to run throttling code

You may want to continue throttling.  I don't recommend sending more than about 150k per day per server, if that, so the added speed is not really necessary.

Regarding the OS, it's CentOS which I'm told is a variant of RedHat 4.4.

I may be missing important differences but to me CentOS, Fedora Core, RHEL, FreeBSD, Solaris, etc. are all very similar.  If you can do something on one you should be able to do, or figure out, the same thing on another.

Mike makes some very good points.  Check your URL as well as your IP with block lists. Re-apply for AOL whitelist and/or feedback loop (I recommend using my script to auto-remove AOL complainants).  Contact abuse@yahoo.com to get a questionnaire that will tell them more about your operation.  If you -really- want to improve delivery you may also want to consider changing your ListMail domain. (Maybe get a .net that corresponds with your .com)  You could even run both off the same database and change it back and forth once a week...  It's a messy solution but possibly effective.

One of the most important things for a dedicated server is the setting up reverse DNS.  Make sure your server hostname resolves to the server IP which in turn resolves back to the server hostname.  To get the reverse IP pointer set up you must contact your server provider.  This could help you avoid a huge delivery problem right off the bat.

Regards

[BGSWebDesign]

Mike, DW,

Thanks for the great tips..

The URL of your LMP is subject to the same standards as any other URL... therefore it could be causing problems. You should check your IP and URL for black listings.

Good point, I'll do that after I move it all over to the new Dedicated server, as soon as I have the new DNS up running properly, actually this isn't too bad, I'm almost there already, and hope to be done before the days over.

On another note... the sawmill might help show initial delivery, but even that in NO WAY guarantees the person actually saw the email or actually even got it. Many mail systems will accept the mail, which to your system seems like it was delivered. But then they process it and run it through checks, blacklists, etc and if it fails just discard it...

Ok, got it, on this - I know I've blasted anyone wanting to check 'delivery rates' using image files in the email, but I wonder if it might be worth it at this point, as it seems sawmill will only show that the email was accepted, and I still won't know if it ever got to anyone.

If not, why do you need to use procmailrc?) If you use signup-by-email I still recommend contacting your host about the possibility of them manually setting up email forwarding to CGI.

Yes, I use signup scripts from the forwarded emails, this works great for me.  Regarding setting up the manual forwarding to CGI - I'll consider it, but now that I'm on dedicated they've already told me I can use "Courier Dot Mail" for email forwarding to scripts - supposedly I may even be able to drop my .procmailrc file into it, but I haven't tested that yet and don't know where to find it, I'll be contacting their tech people heavily the rest of the day until I'm all setup   .

Quote:
I want to remove the 'throttling' code if I can and see if I get more clicks.  

I think throttling can only help delivery, so if all the messages are getting to the server it should only be a few days until the majority of clicks come in. Removing throttling IMHO should not improve things.

No, I've shown previously that throttling even helps get more clicks, because when you're not throttling the system is so busy handling the bounced mail and sending that the clicks increased - this was on shared server, but if you feel it won't do a thing on dedicated, I'll just try to adjust the throttling to get closer to 150K/day which should be no problem, actually I'm getting that on shared server right now, the last broadcast email I sent was running at 2.2/second or about 8,000/hour which is roughly 160,000/day, but I don't think it could handle it continuous because at several points it would just pause for several minutes or more, either that or I hit a limit (on the shared server).

I'll leave the throttling in - but I thought one of the points of running on dedicated is that I could remove the throttling and then get 20,000-30,000/hour out of LMP?

Regarding DNSreport.com that looks amazing, especially this:

This site will provide you with a DNS report for your domain (56 tests are run per DNSreport). A very large percentage of domains have DNS problems; this site will help you find those problems and fix them. Also, the “Mail Test” tool will help find mail delivery problems for your domain.

 

I had no idea anything was available like that, so I'll put that in place too, but first have to get DNS setup, running properly on the dedicated server...  and thanks for the instructions on the Reverse IP, I know that's a problem from previous AOL whitelisting attempts, they wouldn't whitelist me previously because the setup on the shared mail server was not correct.

In my experience AOL will not whitelist you unless you are the sole-sender from your server.

They told me I was whitelisted - using the shared server at my host, that was back in November, 2005, here's the reply email I got from AOL:

Your Whitelist request, with the
confirmation code xxxxxxxxxxx-425774, has been approved.

Anyway, I'm hoping (praying) that this move and dedicated IP/server are going to solve some of the mail delivery and acceptance problems I've had, especially in the last 6-8 months, but even then, the shared server worked fine, so I'm not positive this is going to affect things, I'm a little worried that my list might just be totally unresponsive at this point...  

If you think of anything else let me know - but I'm sure I'll be back posting what I have found, in the next few days...

[BGSWebDesign]

Hi DW, Mike,

Ok, finally I'm rolling on the dedicated server... I still have to make sure the DNS is setup properly and per your suggestion DW I want to make sure the Reverse IP matches the DNS IP, so I'll be working on that as well.  I guess I'll submit that as a ticket to the host company and see what they recommend, unless you have a good idea on how to verify that - can I do it with Ping, or some other method?

So far, something I notice on this new server.  First the Dailymails jammed up again in lm_sendp, so I've had to clean that out and recreate the table.  

One strange thing happened, any ideas on this, I notice my Dailymail report for today shows that 12,678 followups went out last night - but, here's the weird thing, this morning when I logged in to LMP I found that there about 150 leftover messages, I couldn't get them to resume - with resume.php or manually - and I thought I tracked down the problem that Authentication is required by the server - so, how can over 12,000 messages have gone out last night - if LMP was not setup properly, and how could the counter get down to 158 messages left, where did those messages go?  Did they all error out?  Have you heard of that happening before DW?

Then a question about the 'Mail Sending Method' section of Configuration in LMP:  I had to check off Auth? and Authorize by Email to get the emails to start sending with 'Resume', so I wonder, should this Auth email be the Admin email on the Config page, or does it matter?  I think I tried setting it to that, but still could not 'resume', so I set it back to the default email I use - which is really just a system email - it's not used for anything else...  

Is it better for deliverability to MATCH the Auth email with the admin email - or with the email that LMP will do a 'send as email', if I have the same 'send as email' in each list, should I try to match the Auth email to that email address?  Will it help me get delivered, and with validating things from other receipient servers such as AOL, Hotmail, etc?

[DW]

Brett,

I want to make sure the Reverse IP matches the DNS IP. how to verify that - can I do it with Ping, or some other method?

Yes, you can use the linux "host" command.  Here is what success should look like:

Code: [Select]

[root@serv]# host serv.example.com
serv.example.com has address 123.45.67.89

[root@serv]# host 123.45.67.89
89.67.45.123.in-addr.arpa domain name pointer serv.example.com
Of course if you use the "host" command from the server serving DNS for example.com and your namservers don't point to the server the IP returned by test #1 may not be what the world sees... :?

how could the counter get down to 158 messages left, where did those messages go? Did they all error out? Have you heard of that happening before DW?

I have not heard of this happening but every server is different..  There could have been an authentication issue and ListMail missed it or other problem...  Please enable the "always write SMTP log" option immediately so we know for sure whether messages are delivered if this ever happens again.

I had to check off Auth? and Authorize by Email to get the emails to start sending with 'Resume', so I wonder, should this Auth email be the Admin email on the Config page, or does it matter?

What control panel does your server use - cPanel?  Plesk? Other?  On cPanel or Plesk you should be able to send email with "localhost" as the hostname and no AUTH setting.

The "Bounce to email" is used for the "Return-path" and the "Send from Email" for your list is used as the displayed "From" field.  With bounce enabled the Administrator Email is not used in the message, so at best the AUTH should be the bounce account.  Most hosts/servers will allow you to authorize with any account on your domain and then set bounce and "From" to anything on your domain.

Is it better for deliverability to MATCH the Auth email with the admin email - or with the email that LMP will do a 'send as email', if I have the same 'send as email' in each list, should I try to match the Auth email to that email address? Will it help me get delivered, and with validating things from other receipient servers such as AOL, Hotmail, etc?

I don't think so, no.  I think it works just as well to have the bounce email (Return-path) different than the "From" information, although it is highly recommended to have them both be on the ListMail domain.

Regards

[BGSWebDesign]

DW,

I'm having all kinds of problems with my hosting company now - I asked for 'Reverse DNS setup' and they apparently pointed the PTR record to 'thedreamtimemail.com' which they say will NOT work and that it needs to point to 'server.thedreamtimemail.com'!  I'm not sure that will work at all - DW does that make sense to you?  I'm about to pull my hair out with this crap - it's been weeks playing with things, the host techs make mistakes on several occasions costing me hours of wasted 'up-time' and I'm about to go out of business, so I don't have hours and hours and certainly not 'days' to wait for them to figure this out... DW, I'm looking for your input on this:  Is this workable or not???  I'm about to ditch the whole thing and move back to shared server, at least the mail went out, though the delivarability was low - who cares...  

Thanks, I'm almost there...  but I don't see that this is much faster than the old 'shared server', I'd like to relax the throttling a little, here's what I use now, any suggestions on how to determine optimum?

Code: [Select]

if(($y % '100')==0 && $y <> sizeof($sendq)) sleep(10);    


A few funny things happened today. First, because I wanted to test how the delivery of the messages is going, I thought I could go in an change a few records in lm_sendq.  So, I changed the UID in a few records, but I never received those messages, does it work that way DW?

Second - I have 'Always write SMTP log' enabled - I have a ton of files in the Attach folder that are log files - but they are all ZERO Length with nothing in them, what does that mean?   I just did a test mailing to a list with two of my email addresses on it - that seemed to go through fine and it wrote the SMTP log just fine, so what's up with the Zero Length logs?

Third, when I first logged in, LMP was jammed up with over 12,000 followup messages, when I tried to 'resume' with the 'resume' button, I got these errors:

Code: [Select]

Checking mail...
Warning: fsockopen() [function.fsockopen]: php_network_getaddresses: getaddrinfo failed: Name or service not known in /home/bretttdm/wwwroot/mydomain.com/mail/admin.php on line 903

Warning: fsockopen() [function.fsockopen]: unable to connect to thedreamtimemail.com:110 in /home/bretttdm/wwwroot/mydomain.com/mail/admin.php on line 903
Failed!

Connecting to SMTP server...
Warning: fsockopen() [function.fsockopen]: php_network_getaddresses: getaddrinfo failed: Name or service not known in /home/bretttdm/wwwroot/mydomain.com/mail/admin.php on line 954

Warning: fsockopen() [function.fsockopen]: unable to connect to thedreamtimemail.com:25 in /home/bretttdm/wwwroot/mydomain.com/mail/admin.php on line 954
Server not found!
Failed.

Check your SMTP settings!
Your message has been saved for later.

Any idea what's up here?  I had changed the DNS servers, etc, so guessed that might have been it, after working with the tech people, I've reset a few things in the Zone file, and I did a 'restart' on the SMTP Server, after that LMP seemed to be sending as it should, though it took about 3-4 hours to get out 12,000 messages, I'd like this to go a little faster - on the shared server I was getting about 8,000/hour, and I don't see any reason why I couldn't match that on this server - do you, also consider when I test the speed of this server using this - I notice that it's as fast as the Google server: http://www.vertain.com/?sst

Here are the technical specs on the machine:

Code: [Select]

Intel Pentium 1.6GHz
 512MB
 20GB


Other than that, I'm about to setup the Reverse DNS, make sure I get all setup again - White Listed at AOL, check on Hotmail and Yahoo, make sure I can simulate .procmailrc and after that I'm finished with the setup and can continue tweaking...

I guess the thing I'm a little worried about, this server isn't coming anywhere close to the speed/volume I was getting with the shared server...  I'll try setting up the SawMill and see how it's doing, but I'm a little concerned the speed is not matching the 8,000/hour I was getting on shared server??

Any hints/ideas/suggestions/comments welcome on how I can speed this up...