{"_id":"56d553b3c8c687150071efc0","category":{"_id":"56d5537ac8c687150071efbf","__v":6,"pages":["56d553b3c8c687150071efc0","56d553dc075dfa0b00be98f3","56d5582aa4a9211b00c8f21b","56d819277a04df0b00ddf058","56d81a477a04df0b00ddf05c","56d81da4c83c5f0b00d28719"],"project":"56d55013075dfa0b00be98e6","version":"56d55014075dfa0b00be98e9","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-03-01T08:31:54.133Z","from_sync":false,"order":1,"slug":"archiveorg","title":"Archive.org"},"githubsync":"","project":"56d55013075dfa0b00be98e6","user":"56d54fd3c8c687150071efb0","parentDoc":null,"version":{"_id":"56d55014075dfa0b00be98e9","__v":16,"project":"56d55013075dfa0b00be98e6","createdAt":"2016-03-01T08:17:24.051Z","releaseDate":"2016-03-01T08:17:24.051Z","categories":["56d55014075dfa0b00be98ea","56d55036c8c687150071efb3","56d55094b939870b005129e2","56d550ae5acf9f0b00f735d0","56d5537ac8c687150071efbf","573cae191000320e00918252","573cbe15f320ae29009dbd94","5792a0baf967f617000d9e77","5792a0c023106419009c4302","58093f2994bc150f00f2fe47","580940f33c22790f00fd1e98","58094c01e5998d0f00d29b5d","58101442c03f0b0f0097f099","581020f07c59d72500b85add","581032c708d8050f00948d2a","5810e6ca08d8050f00948dcf"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"Archy","version_clean":"1.0.0","version":"1"},"__v":3,"metadata":{"title":"","description":"","image":[]},"updates":["5c4857f2485fc60ff058ada9"],"next":{"pages":[],"description":""},"createdAt":"2016-03-01T08:32:51.028Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":5,"body":"Most archive.org users upload items through the web interface. This option isn't ideal for supporting users who need to upload many items or files at a time. The Internet Archive provides 2 mechanisms for uploading items in bulk. \n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"IAS3 (Experts)\"\n}\n[/block]\nThe first option is to use the ias3 service directly. IAS3 is the Internet Archive's implementation of Amazon's AWS S3 (Simple Storage Service). You can read more about it here <link>. IAS3 provides an API for creating and uploading archive.org items programatically but it requires a high level of technical proficiency to use.\n\nSee common examples at: https://github.com/vmbrasseur/IAS3API/blob/master/examples/README.md\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"curl --location --header 'x-amz-auto-make-bucket:1' \\\\\\n     --header 'x-archive-meta01-collection:opensource' \\\\\\n     --header 'x-archive-meta-mediatype:texts' \\\\\\n     --header 'x-archive-meta-sponsor:Andrew W. Mellon Foundation' \\\\\\n     --header 'x-archive-meta-language:eng' \\\\\\n     --header \\\"authorization: LOW $accesskey:$secret\\\" \\\\\\n     --upload-file /home/samuel/public_html/intro-to-k.pdf \\\\\\n     http://s3.us.archive.org/sam-s3-test-08/demo-intro-to-k.pdf\",\n      \"language\": \"curl\",\n      \"name\": \"upload\"\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"The `internetarchive` Client Library (Everyone)\"\n}\n[/block]\nAnother option is to use the `internetarchive` Python client library. This library includes a command line utility called the \"ia\" tool which wraps IAS3 and makes it easy for anyone to upload in bulk.\n\nThe easiest way to get started, is to use a binary. The only requirements for using a binary are that you have Python 2 or 3 installed in a unix-like environment.\n\nOnce you have [downloaded a binary of ia](http://internetarchive.readthedocs.io/en/latest/installation.html#binaries) and [configured it with your archive.org credentials](http://internetarchive.readthedocs.io/en/latest/quickstart.html#configuring), take some time to familiarize yourself with [what an archive.org item is and what limitations items have](http://internetarchive.readthedocs.io/en/latest/items.html). Then follow these instructions for bulk uploading: http://internetarchive.readthedocs.io/en/latest/cli.html#bulk-uploading.\n\nPlease see http://internetarchive.readthedocs.io/en/latest/metadata.html for more information on item metadata.","excerpt":"","slug":"bulk-upload","type":"basic","title":"Item Bulk Upload"}

Introduction

Archive.org

Wayback Machine

Books

OpenLibrary

Official Client Libraries

Labs IIIF API

Labs / Experiments

Community

Item Bulk Upload

Most archive.org users upload items through the web interface. This option isn't ideal for supporting users who need to upload many items or files at a time. The Internet Archive provides 2 mechanisms for uploading items in bulk. [block:api-header] { "type": "basic", "title": "IAS3 (Experts)" } [/block] The first option is to use the ias3 service directly. IAS3 is the Internet Archive's implementation of Amazon's AWS S3 (Simple Storage Service). You can read more about it here <link>. IAS3 provides an API for creating and uploading archive.org items programatically but it requires a high level of technical proficiency to use. See common examples at: https://github.com/vmbrasseur/IAS3API/blob/master/examples/README.md [block:code] { "codes": [ { "code": "curl --location --header 'x-amz-auto-make-bucket:1' \\\n --header 'x-archive-meta01-collection:opensource' \\\n --header 'x-archive-meta-mediatype:texts' \\\n --header 'x-archive-meta-sponsor:Andrew W. Mellon Foundation' \\\n --header 'x-archive-meta-language:eng' \\\n --header \"authorization: LOW $accesskey:$secret\" \\\n --upload-file /home/samuel/public_html/intro-to-k.pdf \\\n http://s3.us.archive.org/sam-s3-test-08/demo-intro-to-k.pdf", "language": "curl", "name": "upload" } ] } [/block] [block:api-header] { "type": "basic", "title": "The `internetarchive` Client Library (Everyone)" } [/block] Another option is to use the `internetarchive` Python client library. This library includes a command line utility called the "ia" tool which wraps IAS3 and makes it easy for anyone to upload in bulk. The easiest way to get started, is to use a binary. The only requirements for using a binary are that you have Python 2 or 3 installed in a unix-like environment. Once you have [downloaded a binary of ia](http://internetarchive.readthedocs.io/en/latest/installation.html#binaries) and [configured it with your archive.org credentials](http://internetarchive.readthedocs.io/en/latest/quickstart.html#configuring), take some time to familiarize yourself with [what an archive.org item is and what limitations items have](http://internetarchive.readthedocs.io/en/latest/items.html). Then follow these instructions for bulk uploading: http://internetarchive.readthedocs.io/en/latest/cli.html#bulk-uploading. Please see http://internetarchive.readthedocs.io/en/latest/metadata.html for more information on item metadata.