By Rod Suskin on Sunday, 26 January 2014
Posted in Technical Issues
Replies 8
Likes 0
Views 788
Votes 0
Hi

I have set up Amazon S3 remote storage and the files are appearing to transfer correctly. However, I noticed my 'media' folder getting bigger and bigger as users uploaded their pictures.
Upon investigation, I found that the remote storage feature does indeed copy the file to S3, but the local file is not removed. So for example:
1. User #83 Uploaded a picture file of 1MB in size - appears in my server folder /media/com_easysocial/photos/83/155 with an index.html file
2. Some time later, the same folder is created in Amazon S3. In the Amazon folder, the original picture is only 162KB in size. Also in this folder are additional versions of the picture, viz. _thumbnail, _square, _large and _featured.
3. The S3 folder with all the versions totals a little over 250K in size - while the original image at 1MB still sits on my server folder.

In addition, I have discovered that some photographs simply do not get moved to S3 for no clear reason. For example, folder /95 has still not been copied to S3 two days later, while /96, /97 and /98 have been moved.

So what is happening here? What is the point of this remote storage? Why are the original media files the user uploads still sitting there taking up much more space than the remote version? Why are these files not removed from local storage?
That makes sense. Strikes me that you would have to be a magician to get anything to happen instantlhy anywhere on the internet but that will be an amazing feature!
·
Monday, 27 January 2014 00:10
·
0 Likes
·
0 Votes
·
0 Comments
·
Hello Rod,

This is the original idea of the Amazon S3 feature. The reason that the original photo is still stored locally is because we need to rotate images if necessary. If the original photo is stored on Amazon S3, there will be a long delay for the image rotation to work since it needs to first download the 1mb file locally, rotate it and send it back to Amazon S3 again.
·
Sunday, 26 January 2014 20:47
·
0 Likes
·
0 Votes
·
0 Comments
·
Then what is the point of remote storage at all?
·
Sunday, 26 January 2014 21:07
·
0 Likes
·
0 Votes
·
0 Comments
·
Hello Rod,

The main purpose of remote storages is to save on your bandwidth. When there's a lot of visits on your site, it could take up on your bandwidth pretty fast I will see if there's a way we can instantaneously upload to Amazon and also resize images on the fly in the future versions of EasySocial.

In fact, my goal in the near future is to convert the existing remote feature functionality to be able to be done on the fly without even the need of cronjob but there's still a couple of cons that I couldn't find a workaround. Especially when your hosting provider has a bad connection.
·
Sunday, 26 January 2014 22:52
·
0 Likes
·
0 Votes
·
0 Comments
·
Yeah, it's not easy Rod because our main hurdles is on site that is not hosted in the United States or somewhere remote that has a bad connectivity (For instance, where we are at, Malaysia). If we are going to apply remote storage uploads on the fly on a server in Malaysia, a photo upload would easily cost you a minute and it won't make sense
·
Monday, 27 January 2014 01:48
·
0 Likes
·
0 Votes
·
0 Comments
·
What about scenarios such as joomla being hosted on an ec2 instance inside of a VPC. Would the speed not be fast enough?

Can there be an admin option for cron vs hosting the easysocial file system on amazon s3?

It is not an issue of user speed, but more the speed between the computing server and the file storage (s3)?
Sam
·
Monday, 27 January 2014 16:50
·
0 Likes
·
0 Votes
·
0 Comments
·
I think Sam's idea of the option for cron vs hosting on S3 is a great idea.
Also, just to remind: in my case remote storage is extremely buggy - not all photos are transferred, and enabling the option basically breaks the newsstream (I had a ticket about this elsewhere.) I have resolved by disabling remote storage and hoping this will work correctly in 1.2
·
Monday, 27 January 2014 17:53
·
0 Likes
·
0 Votes
·
0 Comments
·
Yeah, I think it makes sense if you are hosted very near or towards area that EC2 is supported but the problem is for users who are located far away like us Making this optional is the most ideal way I believe but I haven't really poked around that yet because it could be a maintenance nightmare upcoming the codes. We'll see what we can do of course
·
Monday, 27 January 2014 18:37
·
0 Likes
·
0 Votes
·
0 Comments
·
View Full Post