My Startup Experience: Developing an Innovative Task-Crawler App
In September 2021, I co-founded a startup with a friend in China to address inefficiencies in the fragmented online task market. This market primarily catered to individuals such as college students and stay-at-home parents, enabling them to earn commissions by completing tasks such as app reviews, content engagement, and live-stream interactions. However, the market was oversaturated with over 300 platforms offering limited tasks, forcing users to frequently switch between platforms and often miss task opportunities due to time constraints and platform-specific schedules.
Our solution was to develop a task-crawler app designed to consolidate these platforms and streamline task acquisition. By integrating APIs from multiple platforms, our app enabled users to claim tasks across platforms simultaneously. Users simply input their account credentials, and the app autonomously logged in, monitored task availability, and claimed tasks in real-time—even while users were engaged in other activities.
My Role in the Project
As the project leader, I was responsible for coordinating the technical team, managing development workflows, and addressing critical challenges throughout the project lifecycle. In addition to overseeing the overall development, I directly contributed by designing and implementing the core crawler functionality using JavaScript. I also proposed innovative technical solutions and designed key user-facing features to enhance the app’s functionality and usability.
Key Technical Solutions
Innovative Crawler Implementation to Save Costs
Innovative Crawler Implementation to Save Costs One of the most significant challenges was integrating over 100 third-party platforms while operating within limited financial and technical resources. Developing platform-specific crawlers using native app languages would have been prohibitively expensive, particularly given the rapid iteration of APIs and the need for frequent updates. To overcome this, I implemented the crawler logic as modular JavaScript scripts that could be dynamically loaded by the app. This approach decoupled the crawler functionality from the app’s core code, enabling seamless updates by simply uploading new scripts to the backend. This eliminated the need to modify or rebuild the app itself. The scripts were initially tested in a Node.js environment to streamline debugging and validation. Once verified, I made minor adjustments to adapt the scripts for integration into the app. This two-phase testing and integration process significantly reduced development complexity and improved efficiency, as testing in a Node.js environment was faster and more flexible than direct in-app testing. This solution not only minimized development and maintenance costs but also provided a scalable and modular approach to platform integration, demonstrating my ability to innovate under resource constraints.
Efficient Integration of Crawler Scripts
Another technical hurdle was enabling seamless communication between JavaScript-based crawler scripts and the app’s core functionalities. To address this, I encapsulated the scripts into modular classes dynamically initialized with user-specific data and app methods. This architecture allowed the scripts to retrieve input data, execute platform-specific tasks, and return real-time results to the app for display. For instance, when a user initiated a task claim on a particular platform, the app initialized the corresponding script class, provided the necessary parameters, and triggered the script to execute. The script then communicated task outcomes or status updates back to the app, ensuring users had a clear and interactive experience. This modular design streamlined the integration process and made the system highly adaptable. Future updates or new functionalities could be implemented by modifying or adding individual scripts without disrupting the app’s overall structure.
Managing Platform Diversity
Each platform integrated into the app featured unique task-claiming logic, authentication protocols, and response formats, such as JSON or HTML. To ensure robust compatibility across these diverse systems, I developed tailored scripts for each platform while simultaneously identifying and encapsulating common functionalities. For instance, many platforms shared similar login workflows or task-claiming mechanisms. By abstracting these shared processes into reusable modules, I significantly reduced development time and ensured consistency across the codebase. For platforms with unconventional response formats, such as HTML requiring specific parsing, I implemented specialized handling within their respective scripts. This systematic approach enabled the app to efficiently manage the diverse requirements of over 100 platforms while maintaining a modular and maintainable architecture.
Performance Optimization
Initially, task-claiming efficiency was suboptimal due to high competition and platform-specific limitations. To address this, I conducted a detailed analysis of platform behaviors and user interactions. One key optimization was adjusting the task-claiming logic to prioritize less-contested tasks further down the task list. This strategy avoided the bottleneck of competing for the most popular tasks and significantly improved claiming success rates. Additionally, users with multiple accounts often experienced diminished efficiency due to excessive requests originating from a single account. I implemented an automated account-switching mechanism to evenly distribute requests across multiple accounts, enhancing task acquisition rates and reducing the risk of account restrictions. These optimizations substantially increased the app’s performance, enabling users to claim tasks more effectively and boosting overall user satisfaction.
User-Centric Design and Features
In addition to technical development, I prioritized user-centered design by incorporating features based on extensive feedback from target users. This approach ensured the app effectively addressed user pain points while providing a superior experience compared to competitors.
Platform Status Monitoring
Unlike competing apps, which offered only basic notifications (e.g., login failures or successful task claims), I designed a platform status monitoring feature that displayed detailed account-related information. For example, if a user’s account was restricted or had unresolved tasks, the app provided clear status updates, enabling users to proactively address the issue rather than encountering unexplained failures. This feature relied on analyzing responses from each platform’s API to extract critical information, which was then presented in a user-friendly interface. This functionality significantly enhanced task acquisition efficiency and provided users with actionable insights.
One-Click Autofill
Managing credentials across multiple platforms—often over 30—posed a significant inconvenience for users. Traditionally, users had to manually input their account details one platform at a time, a repetitive and time-consuming process. After gathering feedback, I discovered that most users relied on one or two sets of account credentials across all platforms. Based on this insight, I designed a one-click autofill feature that allowed users to save multiple credential sets and apply them across selected platforms with a single click. This functionality was implemented through a centralized account management page, where users could store and manage their credentials. When a specific set was selected, the app automatically populated the fields for all designated platforms. This innovation streamlined the onboarding process, saved time, and reduced input errors, providing a significant usability improvement.
Outcome
Following a one-month testing phase, the app rapidly gained traction, with over 8,000 users and 6,000 monthly subscribers within two months. However, in the third month, a competitor entered the market with aggressive advertising, comprehensive platform integration, and free trial promotions, which drew users away. By the fifth month, with only 132 subscribers remaining, we decided to cease operations and refunded the remaining users.
Reflection
Although the startup was short-lived, the experience demonstrated my ability to innovate and solve complex technical challenges under resource constraints. From designing scalable integration solutions to implementing user-centric features, I showcased my problem-solving skills, technical expertise, and adaptability in addressing multifaceted issues.