Subscribed.

Hi,

I am not sure how to assist you in this because this seems like a server issue. Based on the logs that you've provided, there is nothing concrete to go from to be frank.

@Neel: Perhaps it's photo dependent? What I mean is that perhaps the issue happens around the 2mb limit. What is your php.ini set to? Perhaps there is some sort of constraint going on here that is causing this error.

So look into:

- Limits set by your server, Joomla config, and ES settings

- Upload photos one by one. See which ones fail and which ones make it. Check the sizing and such and see if there is a reason why some fail and others don't

Hi Josh, the php limit is not an issue. I've set 32mb as max size for upload size and post size with 256mb memory limit for php scripts. I now noticed that this also happens to some ~ 250kb photos too. Even when multiple photos are added, they are uploaded one by one anyways. But I will try uploading 1 by 1 like you said to see if I can isolate the issue. I will try and more testing and see if I can see a patern to narrow it down on why some fails. Right now like Jason said the errors don't point to anything specific so not sure what's failing where.

Has this happened to anyone else?

Based on the nginx error report, it says "connection reset by peer" which I think it means the server is dropping the connection before the usual 3 way handshake. Usually server has to have a reason why it decided to drop that connection. Another time it says "0 Null primary key not allowed". I don't know what that means, but could it be the reason for server connection reset? Could it be due to the type of photo and the header it has? I don't know....

Everytime I upload something, even if its successful uploads, mysql gives this warning:

141112 13:25:27 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. INSERT IGNORE... SELECT is unsafe because the order in which rows are retrieved by the SELECT determines which (if any) rows are ignored. This order cannot be predicted and may differ on master and the slave. Statement: INSERT IGNORE INTO `joom_finder_terms` (`term`, `stem`, `common`, `phrase`, `weight`, `soundex`, `language`) SELECT ta.term, ta.stem, ta.common, ta.phrase, ta.term_weight, SOUNDEX(ta.term), ta.language FROM `joom_finder_tokens_aggregate` AS ta WHERE ta.term_id = 0 GROUP BY ta.term

Can this be ignored?

I am gonna try increase the request_terminate_timeout and see if that makes a difference. Also is there a way to add a better error handling if uploading photos fails? I mean, if the upload had failed, then a blank photo shouldnt be inserted in the album and ES should perhaps have better error handling in place so empty photos are not added in such scenarios. I found this post in StackOverflow which is similar to what I am experiencing:
http://stackoverflow.com/a/24315338/1050957

Hi Neel,

There is a failsafe actually but the failsafe has to be from the server where the connection still has to be "complete" in order for the server to return a proper error message. Exception thrown on the server side however, means that there is no respond returned from the ajax call, and this can get a bit tricky and we will see if we can add in a better exception in the future.

Thank you Jason. Plz do dig into this to see if you can add a better exception during failure. I wonder if using the sql transaction in innodb could be of use here so if an upload fails it can rollback the last transaction. Even not, there could be another way and I am sure you guys are smart to figure it out.

In the mean time, I analysed all of my configurations starting my php, php-fpm , nginx,and mariadb. Since the main error seemed to be coming from nginx and not php, I am gonna try increasing the timeout settings in my nginx config. I am gonna start by increasing the client_header_timeout, client_body_timeout and keepalive_timeout first and see if that fixes it. Timeout is the only thing I can think of that could cause this issue since the memory limits and upload size seems fine. Do you have any suggestions on a minimum recommended setting when using EasySocial for these 3 plz:

        client_header_timeout 30;

        client_body_timeout 30;

        keepalive_timeout 60;

I am thinking of increasing them to 60, 60, 120.

Hi,

30/30/60 does seem a little too short, especially on uploading large photos, or on internet connection that is not as stable.

Do try 60/60/120 and see how it goes. This is pretty new to us and we will have to figure it out as we go.

Thank you Jason. I have changed the setting to 60/60/120 now and will keep an eye on it. I will report back if the error happens again or if I find something else as the solution. Appreciate your support

I suggest to remove those first or increase the value. As for the client header, I don't think you really need it. Try removing it.

Thank you ken for your input. I'll remove the header timeout then. And increase the body timeout to 60 and keep alive to 120. I don't think keepalive should be causing the issue cuz client can reconnect again. It's most likely the body timeout. Is it safe to remove body timeout too or for the server sake better to have a limit like 60 on it?

Actually thinking about it, I think nginx default is 60 for header and body timeout if my memory serves right. In that case, removing these means it defaults to 60. So probably makes sense to only add them if I wanna increase it to more than 60s.

Actually, I don't suggest any timeouts on the header and body server side since it's a social network.

Cool.. What you say makes sense. I'll remove it and see if it solves the upload issue. Thank you Ken. You are a gem.

All credits to @Jason Rey, can't solve it without him

Hi Jason,

Removing the timeout settings didint make a difference for me. The problem continued to happen. Then I slowly tweaked 1 setting at a time to see if it fixes it but in vain. Later instead of tweaking random settings by hunch, I decided to dive deep into the issue and try and understand what may be really causing this. After hours of analysing basically every services and processes in my server, I found out that the main cause of this HTTP error in my case is due to the OOM-Killer. Basically what is happening here is when the system is running out of memory, the kernel is starting to kill random processes that consumes a lot of memory. It was hard to troubleshoot this cuz sometimes it kills the mysqld process, sometimes the php-fpm and sometimes the nginx process. Also none of these individual services had any proper logs and they just give very vague error due to unknown reasons. But in the end, I found out it is the OOM killer doing all this. I now need to fine tune each services in my server to balance the memory usage and that could hopefully fix this issue. Its a long process and I am not going to be doing this right now since its just a test server. But my production server will have a lot more ram in it and I'll carefully tune it.

During this whole process, what I came to realise is that the photo uploads in easysocial is not handling errors gracefully if such unknown server side error does happen. During the several hours of test I did, I noticed that many failed photos are still showing as empty white boxes in profile and activity streams. For example in 1 album out of 7 photos, 4 of them had HTTP error where the server killed the process during upload and uploading failed. But the album was still showing all 7 photos with 4 of these failed ones in empty white boxes. This is then visible like this for all users to see. So there is a real need for Easysocial to handle this better when such server error occurs so it removes the failed photo from the album and shows a better notice so user can re-upload again. This pain staking error highlighted the real need for error handling when uploading photos. Hope you guys can add something to improve this issue so atleast the client side experience is not compromised when such unknown server side errors occurs..

Hi,

Noted. Will revise on this in the future.