Wednesday, May 6, 2026

Database indexing

Binary search is O(log n) but it only works on sorted data. Suppose you have a customer database and want to find customers whose ages are between 25 and 30. If the database is naturally ordered by customer creation date, you would first need to sort the data by age before performing a binary search to locate the rows corresponding to ages 25 and 30. For a large database, sorting the entire dataset can be time consuming.

A better approach is to create indexes on specific columns. An index on the age column, for example, would contain only two fields: age and customer ID. Because this index is smaller than the full table, it is faster to sort and search. When a query for customers aged between 25 and 30 is executed, the database can perform a binary search (or similar efficient lookup) on the index to find the relevant customer IDs, and then use those IDs to retrieve the full records from the main table.

The downside of indexing is slower write performance. Each time a user creates an account or places an order, the database must update not only the main table but also all associated indexes. Having too many indexes can make write operations, such as clicking a “Save” button, feel slower. Additionally, indexes consume storage space, and at large scale, the total size of the indexes can even exceed that of the actual data.

Tuesday, July 15, 2025

How to write a software issue

If a software problem report or issue describes a large goal or vision, it should be documented as part of a concept of operations or a specification. If you can’t write clear acceptance criteria, or if a developer can’t reasonably complete it in 1–3 working days, it’s probably too big and needs to be split.

Large issues often depend on knowledge of many parts of the system, while smaller ones usually touch just one area. Smaller issues are easier to test, review, and merge. A smaller scope also makes it easier to prioritize and adjust based on feedback. 

Smaller, well-defined tasks are much better candidates as a “first issue” for a newcomer because they feel more approachable. It’s also much clearer when the work is done and what the expected result is. A newcomer can complete the work, submit a pull request, and get feedback quickly — which builds confidence and keeps them engaged.

Monday, June 2, 2025

Non-Functional Requirements

When building software, we define two types of requirements:

  1. Functional: What the system should do (e.g., login, payment).
  2. Non-functional: How well it should do it (e.g., speed, copy protection, security, scalability).

Without this distinction, you risk neglecting important aspects of system quality that don’t show up in features alone. You can implement a feature that works but still fails the user if it’s slow, insecure, or unreliable. Performance is considered non-functional because it affects user experience but doesn’t define what the system does.

It is similar to the frontend-backend layers but not exactly. Frontend-backend is about where things happen. Functional vs. non-functional is about what vs. how well. But there's overlap — non-functional requirements often live in the backend, yet both layers can have them.

In the beginning of a project, non-functional requirements are often forgotten because

  1. They’re abstract and harder to measure
  2. Not visible in UI

But ignoring them leads to rework, failure at scale, and unhappy users. Functional requirements get you a working, minimum viable product. Non-functional requirements make it usable, scalable, and successful.

Wednesday, May 21, 2025

In PHP, "false" can be true

All values loaded from an environment (.env) file using vlucas/phpdotenv are treated as strings, regardless of how they appear in the file. Even if your .env file contains IS_IN_SANDBOX_MODE = false, $_ENV['IS_IN_SANDBOX_MODE'] will be "false". The following condition would evaluate to true, even though the variable in the .env file is set to false, because "false" is a non-empty string and thus truthy in PHP:

if (IS_IN_SANDBOX_MODE) --> if ("false") --> true

This is a common gotcha when working with environment variables in PHP. To address this, you can use:

define('IS_IN_SANDBOX_MODE', filter_var($_ENV['IS_IN_SANDBOX_MODE'],
    FILTER_VALIDATE_BOOLEAN));

Note that FILTER_VALIDATE_BOOLEAN will return null for values like "True", "FALSE", "TRUE", etc. if (null) evaluates to false. It will return true for "true" (lowercase), "1", "on", or "yes". It will return false for "false" (lowercase), "0", "off", "no", or an empty string "".

Alternatively, my personal preference (because it makes the string conversion explicit):

define('IS_IN_SANDBOX_MODE', $_ENV['IS_IN_SANDBOX_MODE'] === 'true');

Or, when using it in a condition:

if (IS_IN_SANDBOX_MODE === 'true')

Wednesday, May 7, 2025

React Native Parameter Passing

When building mobile apps with React Native, we often face a decision about how to pass data between components - especially to screen components. Let's examine two common approaches and when to use each:

1. Direct Props Passing:

interface AddressInfoProps {
    cartItems: CartItem[];
    discountPrice: number;
}
const AddressInfo: React.FC<AddressInfoProps> = ({ cartItems, discountPrice }) => {
    // Component implementation
};

2. Navigation Route Parameters:

import { useRoute, RouteProp } from '@react-navigation/native';
type AddressInfoRouteProp = RouteProp<{
    AddressInfo: {
        cartItems: CartItem[];
        discountPrice: number;
    };
}, 'AddressInfo'>;
const AddressInfo: React.FC = () => {
    const route = useRoute<AddressInfoRouteProp>();
    const { cartItems, discountPrice } = route.params;
    // Component implementation
};

Option 1 is simpler, but it lacks built-in mechanisms to persist state when navigating back and forth between screens. You must always explicitly pass the parameters, meaning the developer is responsible for state persistence. Use it when the component is part of a larger component hierarchy where the parent already has the necessary data to pass down.

Use route parameters (Option 2) when the component represents a screen in your navigation flow—i.e., when backward and forward navigation is expected. With Option 2, the React Native navigation system handles state automatically. If the user navigates to another page and then presses the back button, the screen will re-render with its previous state.

Other options:

Context API: Best for data needed by many components at different nesting levels, but avoid for frequently changing data as it can cause unnecessary re-renders

State Management Libraries (Redux, MobX, Zustand, Jotai): Best for complex applications with lots of shared state and interactions between different parts of the app

URL Parameters: Best for web applications where maintaining bookmarkable state is important

Local Storage: Best for persisting data between app sessions like login status

Monday, May 5, 2025

Storing auth tokens on mobile

React Native AsyncStorage is a simple key-value storage system that saves data as plain text. This means that anyone with access to the device or its backups can potentially read your auth tokens, API keys, or other sensitive data if you use AsyncStorage.

Storing sensitive data unencrypted can also violate Google Play and AppStore guidelines which might result in your app being rejected. Reviewers might run basic security tests that could expose unencrypted token storage.

SecureStore is Expo's abstraction layer that leverages the native security infrastructure of both iOS and Android platforms. SecureStore automatically encrypts all data before storage and provides a unified API that works identically on both iOS and Android. While SecureStore has slightly more overhead due to encryption/decryption operations, the performance impact is negligible for typical use cases like token storage. The security benefits far outweigh any minor performance considerations.

Wednesday, March 5, 2025

Converting a web app to mobile

If you have a responsive web app, you could convert it to a mobile app using React Native WebView without writing any additional mobile code. However, there may be challenges with getting approval from both Google Play and the Apple App Store when using a WebView wrapper approach:
  1. Apple App Review Guidelines Rule 4.2 explicitly states that apps should include features, content, and UI elements that elevate them beyond being just a repackaged website.
  2. Google Play Store: While generally more lenient than Apple, Google still requires apps to provide a reasonably engaging user experience.
The more native integration you include, the more likely you are to pass the review process. Potential Solutions:
  1. Add native functionality such as push notifications or device camera integration.
  2. Implement offline capabilities that aren't available in the web version.
The easiest way to convert your web app to mobile is to ask claude.ai to do it. Your job would be to review, refactor, test and debug the generated mobile code.

Monday, February 10, 2025

Memory cost of image conversion

In my web app, users can upload images as PNG, JPEG etc. I convert them to WebP because it uses much less space on disk and loads faster in web page requests. However, conversion uses raw pixels, so the memory (RAM) required is almost irrelevant to the original file size because file size depends on the compression of raw pixels. The memory needed is primarily determined by: Width * Height * Bytes per pixel. 

A PNG needs 3 bytes for RGB and 1 byte for alpha = 4 bytes per pixel. For a 5637x5637 PNG with RGBA colors, we need 5637 * 5637 * 4 bytes per pixel = ~127MB just for the uncompressed pixel data alone. The file size of that PNG is 6MB.

JPEG files typically don't have an alpha channel, so they need 3 bytes per pixel. For a 5637x5637 JPEG, memory needed would be 5637 * 5637 * 3 = ~95MB. The actual file size of the JPEG could range from:

  • High quality (90%): 2-8MB
  • Medium quality (70%): 1-4MB
  • Low quality (50%): 500KB-2MB
When setting upload limits for image processing, you should primarily consider the image dimensions (megapixels) rather than the file size (megabytes). Here's why:

  1. A user could upload a highly compressed 2MB PNG that's 10000x10000 pixels - this would need ~400MB RAM to process.
  2. Another user could upload a poorly compressed 10MB JPEG that's 1000x1000 pixels - this would only need ~3MB RAM to process.

Considering that current mobile cameras can typically capture 16MP images, the decompressed image would require approximately 16e6 pixels × 4 bytes/pixel / 1024 / 1024 = 61MB of memory. Additional couple of MB memory may be needed as a buffer for libraries like PHP GD, which might use it for copying bytes. 

Here are details from my web app of processing a 16MP JPEG with GD functions (total 172MB, calculated with memory_get_peak_usage):

  • imagecreatefromjpeg: +68MB
  • imagewebp: 130MB (+62MB)
  • resizeWebP:
    • imagecreatefromwebp: 140MB (+10MB)
    • imagecreatetruecolor: 156MB (+16MB)
    • imagewebp: 172MB (+16MB)

To be safe, allocating 200MB of memory would be prudent. If your server has 3GB total RAM, it means that your web app can handle at most 3GB/200MB ≈ 15 concurrent image conversions.

Converting images to WebP saves disk space and speeds up the web app in the long run, but it consumes a lot of RAM in the short run.

Monday, December 23, 2024

Why Software Project Estimation Is Difficult

In my experience, the following factors make software project effort/cost estimation difficult:

  1. It is much easier to estimate for the happy path which leaves gaps in specifications for logging, handling edge/error/timeout cases and exceptions.
  2. It is common to overlook non-functional requirements like performance, security, copy protection, scalability. They often become the reason a project fails at scale, even if it succeeds at first. 
  3. Requirement/design changes due to new information after first user (alpha/beta) tests.
  4. Not taking into account code clean up effort as the project complexity increases.
  5. Changes in libraries and frameworks at least once a year, necessitating unplanned rework.

Every if, switch, or loop condition can introduce new paths through the code. If a function has n independent Boolean conditions, the number of possible paths is up to 2^n. If a function takes multiple parameters, the combinations of edge values increase multiplicatively. 1 input has usually at least 3 edge cases, n inputs would have at least 3^n edge case combinations. If the function depends on or alters internal state, interacts with external systems, or uses asynchronous logic, you must test for edge timing, race conditions, resource exhaustion, etc. Edge cases scale faster than linearly with added logic, often closer to exponential or combinatorial growth depending on interaction depth.

Handling error cases and edge conditions often takes much more effort than the happy path due to the need for additional logic, testing, and debugging. If edge cases are not considered early, addressing them later can introduce costly redesign efforts. Unfortunately, the number and complexity of edge cases grow as the project progresses, especially if new scenarios are discovered during development or testing. Handling one error case might introduce or expose others, creating a chain of additional considerations that were not part of the original estimate.

Neglecting regular code cleanup leads to the accumulation of technical debt, which increases the time and effort required to implement similar features later in the project. New developers joining the team may struggle to understand and contribute to the codebase, further slowing development. Additionally, the effort required to fix bugs, integrate new systems, or perform upgrades can grow exponentially over time.

APIs, libraries and frameworks are typically updated every six months, and programming languages undergo significant changes every few years. While these updates bring improvements, they can impact project timelines.

All these factors can make your initial effort/time estimation 10 times less than the actual cost in the end. A rule of thumb you can use is to estimate effort/time for the happy path and multiply that by 10 to get a realistic number.

Monday, December 2, 2024

Interpreted vs compiled programming languages

With an interpreted language like Python or PHP, if you notice a bug in the web app API logic or the UI script, you can fix it and test the change immediately. This is much faster than stopping to recompile and deploy every time, as would be required in a compiled language.

Without compiled binaries, the same source code can be deployed on multiple platforms without adjustments, as long as the runtime environment is consistent.

Many modern web frameworks (e.g., Flask, Laravel Vite) have features like "hot reloading," which automatically detect changes in code and reload the application without restarting. This is much easier to implement in interpreted environments.

While these advantages are significant, they may come at the cost of runtime performance and error detection, as compilation often catches errors early. However, for most web applications, the increased development speed and flexibility usually outweigh these trade-offs.

Music: The Cardigans - My Favorite Game

Friday, November 29, 2024

Base64 encoding

Base64 encoding ensures that the output only contains characters from a specific, limited set of 64 characters, which are: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

It is safe for most text-based systems because none of the potentially problematic characters (\, \0, \n, \r, \x1a, ', ") appear in the output. Note that \x1a is the hexadecimal representation of the ASCII control character SUB (substitute). It is a non-printable character with the decimal value 26 in the ASCII table. If included in a text string, \x1a is typically invisible and may disrupt processing, especially in legacy systems that interpret it as EOF.

Example:

Base64 encoding increases the size of the input data by approximately 33%. Specifically: For every 3 bytes of input, base64 adds 4 characters. For example if input JSON is {"customer_id":123,"email":"user@example.com","nonce":"d1a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6"}, which is 94 characters, the output will be eyJjdXN0b21lcl9pZCI6IDEyMywgImVtYWlsIjogInVzZXJAZXhhbXBsZS5jb20iLCAibm9uY2UiOiAiZjVkMGMzZGY3ZTM3ODQ2NWQ0NjhkMTdjZTRhNGNlMzIifQ==, which is 128 characters. Note the "==" characters at the end. These are used for padding which ensures the length of the encoded string is a multiple of 4. If your database column to hold the token is VARCHAR(255), assuming a max customer_id of "999 999 999", max email size should not exceed 111 characters.

Email max lengths [RFC 5321, Simple Mail Transfer Protocol]:

  • Local part (before the @): Up to 64 characters (octet = byte).
  • Domain part (after the @): Up to 255 characters.

Total length: The maximum length of a valid email address is 320 characters, but this is extremely rare in practice.

Music: Barış Manço - Dönence

Thursday, November 28, 2024

Concept of Operations

Writing a Concept of Operations (CONOPS) is the first and most important step in a project and should ideally be prepared before a contract is signed. The CONOPS provides a shared understanding of the project's goals, scope, and operational context. This clarity is essential for creating a contract that accurately reflects stakeholder expectations. It bridges the gap between stakeholders' needs and the technical execution. It includes example workflows of at least the happy paths to demonstrate how the system will function. These workflows should clearly specify who is doing what; in other words, there should be no sentences written in the passive voice. Example workflow for email verification on new customer registration:
1. The customer clicks the "Register" button on the registration form.
2. The browser sends a registration request to the server.
3. The server generates a customer email verification token.
4. The server saves the token to the email_verification_token field in the database.
5. The server sends an email to the customer containing the verification link with the token.
6. The server responds with the page: “Registration successful. We have sent you an email. Open that email and click on the verification link to complete the registration.”
7. The browser displays the "Registration successful" page to the customer.
8. The customer opens the email and clicks the verification link.
9. The browser sends a verification request to the server.
10. The server retrieves the customer from the database using the token included in the request.
   a. If the token cannot be found, the server responds with the login page.
11. The server sets the customer's email_verification_token field to NULL.
12. The server responds with the page: “Registration complete. Link: Login”
13. The browser displays the page to the customer.
14. The customer clicks the login link.
15. The browser sends a login request to the server.
16. The server retrieves the customer's data for the login email.
17. The server verifies the password. If the password is correct, the server checks the email_verification_token field. 
18. If the token is NULL, the server logs in the user and responds with the customer dashboard page.
    a. The browser displays the dashboard to the customer.
19. If the token is not NULL, the server responds with the page: “To continue with the login, click on the verification link in the email we have sent you. Link: Send verification email again.”

Sunday, November 17, 2024

Data Transfer Objects

Data Transfer Objects (DTOs) decouple the internal structure of your models, entities, or database schema from the data exposed to the external world (e.g., APIs, front-end). They serve as a contract for the data being transferred, which helps ensure that changes in your data layer don't inadvertently affect the consumer.

Another benefit of DTOs is to enforce structured data and eliminate errors from missing [string] keys. By using a constructor, you control how the raw data is mapped. If a key is missing, you can provide a default value or handle the absence gracefully or you can add validation logic centrally in constructor. Using DTOs allows tools like PhpStorm or VSCode to identify missing properties during development. When working with arrays, typos in keys can go unnoticed until runtime, with DTOs you are using object fields which don’t have this weakness. Example without DTO (note the usage of error prone string keys):

private function prepareProductDetails(array $product): array {
        $product['photos'] = $this->processProductPhotos($product['photos']??[]);
        $product['tr_currency_formatted'] = isset($product['deposit']) 
                ?Currency::formatTRY($product['deposit'])
        return $product;
    }

With DTO:

class ProductDetailsDTO {
    public array $photos;
    public ?float $deposit = null;
    public ?string $tr_currency_formatted = null;
    public function __construct(array $data) {
        $this->photos = $data['photos'] ?? [];
        $this->deposit = $data['deposit'] ?? null;
    }
    public function setFormattedCurrency(callable $formatter): void {
        if ($this->deposit !== null) {
            $this->tr_currency_formatted = $formatter($this->deposit);
        }
    }
}

Note the usage of object fields instead of string keys:

private function prepareProductDetails(array $product): ProductDetailsDTO {
    $productDTO = new ProductDetailsDTO($product);
    $productDTO->photos = $this->processProductPhotos($productDTO->photos);
    $productDTO->setFormattedCurrency(fn($amount) =>
        Currency::formatTRY($amount));
    return $productDTO;
}

Saturday, October 19, 2024

Using UI with AI

If your web or mobile app has multiple user interface (UI) commands, (such as log in, register, search, show products, change user settings), users might struggle to know exactly where to click. The UI would be much more user-friendly if an AI could interpret user speech and convert it into commands that can be handled by the backend. Today’s AI is robust enough to map different phrases that mean the same thing to a single command. For example, a user might say "register" or "create a new account," and both can be mapped to the command "sign_up." The AI can understand both English and Turkish, for example "bana yeni bir kullanıcı oluştur" correctly maps to "sign_up". Here is a demo in Python:
When you use an API, such as OpenAI, the main disadvantage is that you must pay for every API call. Therefore, using voice commands to control the UI should be limited to paying customers, and there should be rate limits in place to keep costs under control. You might use open-source models like LLaMA to run the AI on your own server, but that would require better computational and memory resources than you currently have.

17.02.2025: Open source AI models like DeepSeek open the door to self hosted AI. You will need a powerful server with lots of RAM and GPU. The key will be maximizing GPU VRAM, as this is the primary bottleneck for running large models. For efficient inference, the entire model (or as much of it as possible) needs to reside in VRAM. Such servers might cost more than AI API calls if you use a cloud server. One solution might be to have your own physical server to run the AI model and use the cloud server for the web app, which makes API calls to the AI on your server. You will also need to host a speech to text model.

19.06.2025Desktop AI Compared

Sunday, October 13, 2024

Evaluating fairness of an Investment/Shareholders' Agreement

When an investor invests in a startup, they present the company with an Investor/Shareholder Agreement, which outlines the rights and obligations of the shareholders within the company, including voting rights, share transfer procedures, dividend policies, and protections for minority shareholders. These agreements safeguard the investor's financial interests and govern their relationship with the company and other shareholders.
It is common for the initial investor to have more rights. Normally, early investors expect the following protections:
  • Board representation (a seat on the board).
  • Veto rights on major financial decisions (e.g., capital raises, mergers, or asset sales).
  • Approval rights over key hires or changes in the business direction.
  • Liquidation preferences to get paid first in case of a company exit.
But sometimes their demands can be excessive. To evaluate the fairness of such agreements, you can upload the proposed agreement to chatGPT and use the following prompt:
Does the contract reflect a balanced distribution of power? If not, what share percentage would correspond to the class B shareholder's power? Is this normal for an initial investor in the startup, considering that convincing the first investor is often the most difficult?
If the agreement grants a 15% share in the company but provides 50% control, which is more aligned with a controlling or near-majority stake, we can say the agreement is not fair. Common unfair clauses in such agreements are:
  • Even though the class A shareholder (founder) appoints the board member, any change in the board representative requires class B shareholder (investor) approval.
  • Class A shareholder cannot transfer shares without class B consent for 3 years, while class B share holder is free to transfer to affiliates or related parties without restriction.
  • Important strategic decisions need approval from the Steering Committee, where both A and B share holders have one representative each. Any deadlock in this two-person committee could give the B shareholder veto power over important business decisions​.
You can use the following questions to persuade the investor to be more flexible:
  • Do we agree that the founder/CEO of an ambitious startup with rapid growth goals needs to be able to act quickly, requiring minimal approval/bureaucracy?
  • To fund rapid growth, we most probably will need other investors. Can we foresee that this agreement might irritate potential investors, lower the company's value, and lead them to request the same privileges? 
  • If the same privileges are granted to other investors, would reaching an agreement on any matter outside of routine business—especially considering the potential for irrational behavior (such as ego conflicts, etc.)—become practically impossible?
The goal is to help the investor see the mutual benefits of more flexible terms. You want to highlight that while protecting their investment is important, collaboration, agility, and attracting future investment will ultimately lead to better outcomes for both parties.

Sunday, September 22, 2024

What tech to use when you are just starting

In the beginning, you are trying to find out if there is a need for your app, which means you personally have to find people to try your app. Since there is no marketing, there is no sudden spike in traffic you have to think about. Therefore, you should aim for quick prototyping instead of scalable solutions. Here is a list with the complex option in the first column and simpler option in the second:
# complex simple
1 Mobile app Progressive Web App (PWA)
2 MySQL/PostgreSQL SQLite
3 Queue worker defer()
4 Broadcast with events Echo whisper
5 Mailgun PHPMailer + gmail
6 ElasticSearch SQL queries
The simpler options also use less resources which means that you can host your app on a cheap VPS for 10$/month.

Friday, September 20, 2024

The role of the CTO

The role of the Chief Technology Officer (CTO):

  1. Understand business goals, assess feasibility, and set technical priorities accordingly. Create a technical roadmap.
  2. In the early stages of the company, be the lead engineer. As the company grows, hire technical personnel, provide guidance to the team.
  3. Review team outputs (code & documentation), ensure effort is aligned with priorities and outputs are clean and of high quality.
  4. Review tests and verify that important test cases are included.
  5. In case of personnel loss, temporarily fill the gap until a replacement is hired.
  6. Oversee IT, CI/CD infrastructure and security.
  7. Stay updated with technological trends.
  8. Represent the company's technological vision to partners, investors, or at industry events.

Thursday, September 19, 2024

Financial projection for startups

If you have a startup and are seeking investment beyond friends and family, you have to offer a product or service with exponential growth potential. Given that most startups fail, linear growth doesn’t justify the risk for investors. Your investors would also like to see your revenue estimates, i.e. financial projection/forecast.

Once you can genuinely convince yourself and others of your startup's exponential growth potential, the next step is to estimate the total addressable market and your projected share of it 12 months after launching your product and beginning aggressive marketing. You should also make a realistic revenue estimate for the first month. Assuming a logistic growth pattern—characterized by rapid early growth that slows as it nears a saturation point—you can then project revenue for each subsequent month.

For example, if revenue at product launch + 1 month is estimated as 30K$, and revenue at +12months as 5M$, the logistic function becomes (n: month):

Using this equation, you can calculate revenue targets for each month:

Friday, August 30, 2024

Why use SQLite?

SQLite is optimized for situations where the workload is predominantly reads with occasional writes, making it ideal for small to medium-sized applications. SQLite is very efficient and can operate comfortably within a few megabytes of RAM.

For highly concurrent environments where many users need to write to the database simultaneously, SQLite might not be the best choice, and a more robust client-server database like MySQL or PostgreSQL would be more appropriate. Recommended RAM for light use for MySQL is 1GB and for PostgreSQL 2GB, which is about 1000 times more than SQLite.

SQLite is an embedded database that runs in the same process as the application using it. This means it doesn’t require a separate server process to manage database connections, unlike MySQL, which operates as a server and listens for incoming connections on a specific port (e.g. 3306). SQLite databases are stored as files on the local filesystem. When an application wants to interact with an SQLite database, it directly accesses the database file without needing to communicate over a network. So with SQLite, if you run multiple apps, there won't be problems like port conflicts or the need for containers.

SQLite doesn't require authentication mechanisms like usernames and passwords. The database file is accessed directly without any user credentials.

If you want to use a cheap (5$/month) VPS with 512GB RAM, SQLite is your only option. It is not just a toy, it is used in popular apps like nomadlist, see Pieter Levels video. You can optimize SQLite even further.

Music: Mike Oldfield - Sentinel

Friday, August 23, 2024

Web server vs database server scaling

Every user interaction with a website, such as loading a page, clicking a button, or submitting a form, generates a request to the web server. These interactions are frequent and often involve static assets (like images, CSS, and JavaScript files), rendering HTML, and processing logic. As the number of users increases, the web server has to handle a rapidly growing number of these requests.

While many web requests may involve querying the database, they don't always result in database interactions. For example, pages with static content, content cached in memory, or content generated by the web server without requiring a database query won’t proportionally increase the database load. Additionally, many web requests might use the same data that can be cached on the web server, reducing the number of database queries.

Only dynamic content that requires data retrieval or storage will result in database queries. Since not all web interactions require new data from the database, the database load doesn't increase as rapidly as the web server load.

Initially, it's easier and more effective to scale web servers horizontally to handle increased traffic. The database server can handle a significant amount of load on its own due to its ability to manage data consistency, and because you can optimize performance through caching and other techniques. Once the traffic and data operations reach a certain threshold, you will need to consider scaling your database server as well.