Database indexing

Binary search is O(log n) but it only works on sorted data. Suppose you have a customer database and want to find customers whose ages are between 25 and 30. If the database is naturally ordered by customer creation date, you would first need to sort the data by age before performing a binary search to locate the rows corresponding to ages 25 and 30. For a large database, sorting the entire dataset can be time consuming.

A better approach is to create indexes on specific columns. An index on the age column, for example, would contain only two fields: age and customer ID. Because this index is smaller than the full table, it is faster to sort and search. When a query for customers aged between 25 and 30 is executed, the database can perform a binary search (or similar efficient lookup) on the index to find the relevant customer IDs, and then use those IDs to retrieve the full records from the main table.

The downside of indexing is slower write performance. Each time a user creates an account or places an order, the database must update not only the main table but also all associated indexes. Having too many indexes can make write operations, such as clicking a “Save” button, feel slower. Additionally, indexes consume storage space, and at large scale, the total size of the indexes can even exceed that of the actual data.

Memory cost of image conversion

In my web app, users can upload images as PNG, JPEG etc. I convert them to WebP because it uses much less space on disk and loads faster in web page requests. However, conversion uses raw pixels, so the memory (RAM) required is almost irrelevant to the original file size because file size depends on the compression of raw pixels. The memory needed is primarily determined by: Width * Height * Bytes per pixel. 

A PNG needs 3 bytes for RGB and 1 byte for alpha = 4 bytes per pixel. For a 5637x5637 PNG with RGBA colors, we need 5637 * 5637 * 4 bytes per pixel = ~127MB just for the uncompressed pixel data alone. The file size of that PNG is 6MB.

JPEG files typically don't have an alpha channel, so they need 3 bytes per pixel. For a 5637x5637 JPEG, memory needed would be 5637 * 5637 * 3 = ~95MB. The actual file size of the JPEG could range from:

• High quality (90%): 2-8MB
• Medium quality (70%): 1-4MB
• Low quality (50%): 500KB-2MB

When setting upload limits for image processing, you should primarily consider the image dimensions (megapixels) rather than the file size (megabytes). Here's why:

1. A user could upload a highly compressed 2MB PNG that's 10000x10000 pixels - this would need ~400MB RAM to process.
2. Another user could upload a poorly compressed 10MB JPEG that's 1000x1000 pixels - this would only need ~3MB RAM to process.

Considering that current mobile cameras can typically capture 16MP images, the decompressed image would require approximately 16e6 pixels × 4 bytes/pixel / 1024 / 1024 = 61MB of memory. Additional couple of MB memory may be needed as a buffer for libraries like PHP GD, which might use it for copying bytes. 

Here are details from my web app of processing a 16MP JPEG with GD functions (total 172MB, calculated with memory_get_peak_usage):

• imagecreatefromjpeg: +68MB
• imagewebp: 130MB (+62MB)
• resizeWebP:
• imagecreatefromwebp: 140MB (+10MB)
• imagecreatetruecolor: 156MB (+16MB)
• imagewebp: 172MB (+16MB)

To be safe, allocating 200MB of memory would be prudent. If your server has 3GB total RAM, it means that your web app can handle at most 3GB/200MB ≈ 15 concurrent image conversions.

Converting images to WebP saves disk space and speeds up the web app in the long run, but it consumes a lot of RAM in the short run.

Base64 encoding

Base64 encoding ensures that the output only contains characters from a specific, limited set of 64 characters, which are: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

It is safe for most text-based systems because none of the potentially problematic characters (\, \0, \n, \r, \x1a, ', ") appear in the output. Note that \x1a is the hexadecimal representation of the ASCII control character SUB (substitute). It is a non-printable character with the decimal value 26 in the ASCII table. If included in a text string, \x1a is typically invisible and may disrupt processing, especially in legacy systems that interpret it as EOF.

Example:

Base64 encoding increases the size of the input data by approximately 33%. Specifically: For every 3 bytes of input, base64 adds 4 characters. For example if input JSON is {"customer_id":123,"email":"user@example.com","nonce":"d1a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6"}, which is 94 characters, the output will be eyJjdXN0b21lcl9pZCI6IDEyMywgImVtYWlsIjogInVzZXJAZXhhbXBsZS5jb20iLCAibm9uY2UiOiAiZjVkMGMzZGY3ZTM3ODQ2NWQ0NjhkMTdjZTRhNGNlMzIifQ==, which is 128 characters. Note the "==" characters at the end. These are used for padding which ensures the length of the encoded string is a multiple of 4. If your database column to hold the token is VARCHAR(255), assuming a max customer_id of "999 999 999", max email size should not exceed 111 characters.

Email max lengths [RFC 5321, Simple Mail Transfer Protocol]:

• Local part (before the @): Up to 64 characters (octet = byte).
• Domain part (after the @): Up to 255 characters.

Total length: The maximum length of a valid email address is 320 characters, but this is extremely rare in practice.

Music: Barış Manço - Dönence