Wednesday, July 30, 2014

How to download data from Coursebuilder

1. Set up Google App Engine

Firstly, the assumption is that you have downloaded and installed the GoogleAppengine launcher and coursebuilder. You may get this image:


2. Implement the global environment variable for running etl.python

Secondly, if we want to download or upload data, we need use /tools/etl/etl.py. In order to run etl.py, we firstly need to set the environment, so we need create a file named gcbpaths.py and add the following code to gcbpaths.py:

import sys
// the only change for us -- put the directory where coursebuilder is stored
sys.path.insert(0,'/Users/haoyuchen/Developer/Mooc/coursebuilder-07-29/')

sys.path.insert(0,'/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine')
sys.path.insert(0,'/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/fancy_urllib')
sys.path.insert(0,'/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/jinja2-2.6')
sys.path.insert(0,'/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2')
sys.path.insert(0,'/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webob-1.2.3')
sys.path.insert(0,'/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/yaml/lib')

import argparse
import etl
if __name__ =='__main__':
    PARSER = argparse.ArgumentParser(add_help=False, parents=[etl.PARSER])
    etl.main(PARSER.parse_args())

Notes: This is the exactly directory for etl, you can't miss it

3. Understand input for running gcbpaths.py

Thirdly, We will try to download the course material data. The command line would be:
python tools/etl/gcbpaths.py download datastore <course_url_prefix> <app_name> <app_server_name> --archive_path <archive_path> --datastore_types <model1,model2>

Real example: 
python tools/etl/gcbpaths.py download course /tapestryworkshop mycourse0724 mycourse0724.appspot.com --archive_path ./tapestryworkshop.zip

Command description:
action : download
data type:  course 
course_url_prefix: /tapestryworkshop
app_name: mycourse0724
app_server_name: mycourse0724.appspot.com
archive_path:  ./tapestryworkshop.zip



After run the command, then you need fill your gmail and password.

Example  1 --Download event entity data 
There are two data types .
1. course  --  All is about course information
2. datastore -- All is about the user information (included the watching behaviors!)

In order to monitor the event entity data, we need set download data type as datastore. And we also need set the related tag to be true. If not, we could just get small amount of data.

Steps to turn on the event tag monitors:
1. Go to the course admin page as following.
2. Click setting here and select these three tag to be active:

3. Set the true tag to be true and active, then remember to save:

4. run python code: For instance, 
python tools/etl/gcbpaths.py download datastore /mediatesting lighthouseprojectmooc lighthouseprojectmooc.appspot.com --archive_path ./6_06_9.enventenity.zip



Then we could get event entity data from the datastore:
On the other hand, the entity data from course material:


Example 2 -- Download course from online then upload it to another online server.
Steps:
1. Download course data
The course we want to transfer:

Command:
python tools/etl/gcbpaths.py download course /tapestryworkshop mycourse0724 mycourse0724.appspot.com --archive_path ./tapestryworkshop.zip

result:
Download result -- tapestryworkshop.zip

2. Upload course data to another server(use local server as example)
from GoogleAppEngineLauncher, it shows the local server's ID is chenhaoyu1992, app_server_name is localhost:13080

Before uploading, we need firstly create a new empty course:
The context path (/new_course) will be used as course_url_prefix(one of input parameters for etl.py).
New Empty Course

     


Command:
python tools/etl/gcbpaths.py upload course /new_course chenhaoyu1992 localhost:13080 --archive_path ./tapestryworkshop.zip

Command description:
action : upload
data type:  course 
course_url_prefix: /new_course
app_name: chenhaoyu1992
app_server_name: localhost:13080
archive_path:  ./tapestryworkshop.zip

After uploading successfully, we could run it in our local server.

New course with uploaded content
Important:
But one thing should be remembered is that we could upload course data to a new server. But now we couldn't upload the student's data into the server. Because the google team doesn't implement that code.













No comments:

Post a Comment