The Power of AGGREGATION in Cron Jobs and Cost-Effectiveness

While working on my SaaS product I found, For 10k users, you'd need 10,001 queries daily with regular DB queries to reset credits or free prompt. With smart aggregation, you only need 2 queries, no matter if you have 10k or 100k users!

Firstly, let me give you some COST REVIEW for MongoDB production database (10k & 1 year):

Normal way, Daily Queries: 10,001
Annual Queries: 10,001 x 365 = 3,650,365 queries
Annual Cost: 3,650,365 x $0.001 = 3,650.37 USD

Aggregation way, Daily Queries: 2
Annual Queries: 2 x 365 = 730 queries
Annual Cost: 730 x $0.001 = 0.73 USD

Savings: 3,650.37 - 0.73 = 3,649.64 USD (nearly 4 lakh bdt)

Awesome, now look at traditional approach of query (which make one query for each user )

const resetLimitsForUsers = async () => {
  const users = await User.find({ /* conditions to select users */ });

  for (const user of users) {
    if (user.plan.remaining_prompt_count < 3 || user.plan.remaining_page_count < 3) {
      user.plan.remaining_prompt_count = 3;
      user.plan.total_prompt_count = 3;
      // Save updated plan
      await user.plan.save();
    }
  }
};

Here if you have 10,000 users, this results in 10,001 queries (1 for each user, plus the initial query to fetch users) - that was huge..

Now the hero entry, [ which looks little tough but it saves tons of your money ]

const resetPlanCounts = () => {
  cron.schedule('* * * * *', async () => {
    try {
      const twoMinutesAgo = new Date(Date.now() - 2 * 60 * 1000); // 2 minutes ago

      const usersWithRegisteredPlan = await User.aggregate([
        {
          $match: {
            createdAt: { $lte: twoMinutesAgo },
            plan: { $exists: true }
          }
        },
        {
          $lookup: {
            from: 'plans',
            localField: 'plan',
            foreignField: '_id',
            as: 'planDetails'
          }
        },
        {
          $unwind: '$planDetails'
        },
        {
          $match: {
            'planDetails.name': 'Registered',
            $or: [
              { 'planDetails.remaining_prompt_count': { $lt: 3 } },
              { 'planDetails.remaining_page_count': { $lt: 3 } }
            ]
          }
        },
        {
          $project: {
            planId: '$planDetails._id'
          }
        }
      ]);

      const planIds = usersWithRegisteredPlan.map(user => user.planId);

      if (planIds.length > 0) {
        const { modifiedCount } = await Plan.updateMany(
          { _id: { $in: planIds } },
          { $set: { remaining_prompt_count: 3, total_prompt_count: 3, remaining_page_count: 3, total_page_count: 3 } }
        );

        console.log(`${modifiedCount} plans reset for "Registered" users.`);
      } else {
        console.log('No plans to reset for today.');
      }
    } catch (error) {
      console.error('Error resetting plan counts:', error);
    }
  });
};

That's how you can run your cron job [ it runs automatically in a specific time ] for updating all 10k users credits or limit which can save more than 3600 USD in a year.

AUTHOR,
Name: Mahinur Rahman
Contact: dev.mahinur.rahman@gmail.com

The above is the detailed content of The Power of AGGREGATION in Cron Jobs and Cost-Effectiveness. For more information, please follow other related articles on the PHP Chinese website!