I’m writing a liveblogging app for sports events. It consists of two main DIVs; one is the list of updates from the event, and the other other is going to be a chat window where users can sign in, either through the website itself, or Facebook, or Twitter, and then contribute their own comments.
I ran a live test of my app today and it failed on an epic scale. Server load went through the roof and the entire box became completely unresponsive. I had to power down and eventually get a terminal session on the server to stop the web server (nginx) and php5-fpm. In the end we went back to the current version of the software which doesn’t have comments built-in and successfully covered the event.
Both the chat DIV and the list of updates are dynamically updated by using jquery to do an ajax hit to the server every four seconds; it hits an action on the server, passing the ID of the most recent message, and the server returns new messages (if any) which are then formatted and prepended to the existing list. This obviously results in an increased number of hits on our server, and it’s a relatively high traffic site as it is, but not so many that I thought it would knock it over completely.
So, I’m looking for advice as to how I might make it more performant. I’m thinking I could:
Abandon a framework and write the code myself, although I’d really quite like to use a framework since I’m not a PHP expert and I’d like to be encouraged into doing things well - and if you Google ‘high performance php framework’, Yii comes up number one!
Throw hardware at the problem. This runs on an AWS server, so it’s no inconceivable that I could just fire up additional web server instances. This feels like a generally bad solution though - the current (hand-rolled) version of the software doesn’t have anything like these performance problems, and while I appreciate I’m adding a load more hits by incorporating a chat application, it doesn’t feel like I’m making ridiculous demands of what we have.
Use a Javascript publish/subscribe system, such as Pubnub, to handle the message traffic (at least for the comments). This is pretty much a no-brainer, and although I’d like to avoid using any service that will charge me, I’m not sure I have an option.
I realise these aren’t exactly Yii-specific questions, but if you could suggest ways in which I could make Yii run faster, or not seem to use quite so much load, I’d be really grateful.
Hi Alex, thanks for that. I’m embarrassed to say that no, I hadn’t been through that, so thanks for the pointer. Heh, running in debug mode. Gah, that’s silly stuff!
Not sure I can really do much to optimise the database - it’s pretty small scale stuff, with only a few tables, although I guess there are a few joins going on here and there. I’ve set the schema caching duration to be 24 hours (although in a production ready app, I’m not sure there’s many times you wouldn’t want to set that to always cache, and then just change it when you do a release?).
Re caching generally, that’s a bit difficult given the nature of the application (are you familiar with The Guardian website’s minute-by-minute, or live blogs - it’s that idea) but I’ll see if there’s anything I can do with memcache or something.
With regards to the load it got, looking back at the logs we got from the old software, it had about 20k unique visitors in the course of 90 minutes, which I suppose is quite a lot to ask of my bog standard cloud server.
The problem I have now is that without publishing the URL for another event, I don’t really have a way of sufficiently capacity testing. Do you know of any load balance test sites? I did run this through loadimpact.com’s tests, but it passed easily, so I’m not sure they’re a great test for this site in real terms.
Not really speaking to the specific solution as much as normal troubleshooting, but have you tried breaking the problem down and identifying where you’re seeing the biggest slowdown?
Latency on the client?
Latency in transport?
Latency in the SQL Query?
As Alex suggested, could you query the server every 4s and cache those query results, so that anytime someone hits it in the 4s span, each user will get identical results (even if something was posted 1s ago)? This would reduce any latency that lives in the query (ie, the multiple joins you talk about).
Is the formatting of the posts you’re passing back a lot of overhead? Could you leverage client-side script to do the format instead?
You say you’re pre-pending it to a list…is this a physical list stored in memory, or is it just modifying the display components to have some items before the existing ones? (if you’re reordering an array, it could be costly)
Sounds like the scale of your application is such that you may benefit from some outside help with capacity testing…do you have a QA environment? Have you set up any automated tests? You’ll probably want to create a service that can create posts randomly under multiple sessions, and one that consumes the main page…neither of these are really good substitutes for a proper load test, but might give you some clues to where you should be focusing effort.
