UNPKG

@turingnova/robots

Version:

Next.js robots.tsx generator - Automatically create and serve robots.txt for Next.js applications

132 lines (90 loc) ā€¢ 3.23 kB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
# Quick Start Guide ## šŸš€ Install and Get Started ### Step 1: Install the package ```bash npm install @turingnova/robots ``` **āœ… Files are automatically created:** - `robots.tsx` - For Next.js applications - `robots.txt` - For traditional web servers ### Step 2: Serve robots.txt on your domain **Choose one of these options:** #### Option A: Simple Server (Recommended for testing) ```bash npx robots-serve ``` - Visit: https://example.com/robots.txt - Your robots.txt will be served correctly #### Option B: Next.js Integration ```bash # Move robots.tsx to your app directory mv robots.tsx app/robots.tsx # Next.js will serve it at your-domain.com/robots.txt ``` #### Option C: Traditional Web Server ```bash # Move robots.txt to your public directory mv robots.txt public/robots.txt # Your web server will serve it at your-domain.com/robots.txt ``` ### Step 3: Customize your robots.txt **Choose one of these methods:** #### Method A: Web Editor (Visual) ```bash npx robots-web # Open http://localhost:3000 # Edit visually and save ``` #### Method B: Direct Edit ```bash # Edit robots.txt directly nano robots.txt # or code robots.txt ``` #### Method C: CLI Commands ```bash # Generate with custom settings npx robots generate --sitemap https://yourdomain.com/sitemap.xml ``` ## šŸ”§ Common Issues & Solutions ### Issue: "robots.txt not found on my domain" **Solution:** The robots.txt file needs to be served by a web server. 1. **For testing:** Use `npx robots-serve` 2. **For Next.js:** Move `robots.tsx` to `app/robots.tsx` 3. **For traditional servers:** Move `robots.txt` to `public/robots.txt` ### Issue: "How do I customize the content?" **Solution:** Use the web editor or edit files directly. ```bash # Web editor (recommended) npx robots-web # Or edit directly nano robots.txt ``` ### Issue: "How do I deploy to production?" **Solution:** Choose your deployment method: 1. **Next.js:** Deploy normally, robots.tsx will be served automatically 2. **Static hosting:** Place robots.txt in public folder 3. **Custom server:** Add robots.txt serving to your server ## šŸ“‹ Checklist - [ ] Package installed: `npm install @turingnova/robots` - [ ] Files created: `robots.tsx` and `robots.txt` exist - [ ] robots.txt served: Accessible at your-domain.com/robots.txt - [ ] Content customized: Updated for your specific needs - [ ] Deployed: Working in production ## šŸŽÆ Success Indicators āœ… **robots.txt accessible at your-domain.com/robots.txt** āœ… **Content is appropriate for your site** āœ… **Search engines can read your robots.txt** āœ… **No additional commands needed after install** ## šŸ’” Pro Tips 1. **Use the web editor** for easy customization 2. **Test with robots-serve** before deploying 3. **Update sitemap URL** to your actual sitemap 4. **Customize disallow paths** for your specific needs 5. **Use Next.js robots.tsx** for better SEO integration ## šŸ†˜ Need Help? - **Web Editor:** `npx robots-web` - **Simple Server:** `npx robots-serve` - **CLI Help:** `npx robots --help` - **Documentation:** See README.md for full details