CMSE802
0907--AvsN-pre-class-assignment
Where you able to clone the git repository for this course? If not, where did you get stuck?
Easy/Hard parts
Calculus
Pretty well - it was good review for a guy who hasn't had calculus in 20+ years.
Git Pull Error
MacBook-Pro:~ user$ git pull https://gitlab.msu.edu/cmse802-f20/cmse802-f20
fatal: not a git repository (or any of the parent directories): .git"
SSH keys in gitlab
I think I'm all good! The only resource I pulled in was this document from Gitlab on how to generate an ssh key. It was really useful if you want to use it for future students!
https://gitlab.msu.edu/help/ssh/README#generating-a-new-ssh-key-pair
Other Questions
Is there a preference for where we should be running the class files (ie. on our own computer or on the command line in GitLab)?
Other questions
Can we do a live tutorial or how to use git commands like pull and status in class?
I hope we could have some git demo in the future class.
0909-Optimization
Where you able to get the optimization library to work for the simple example?
No I couldn't figure out what method to use or what further information I could I have fed the minimize function to get to run. The documentation was a little hard to figure out and internet examples solved problems a bit more complex than the example we used.
I tried to solve for the local minimum with scipy optimization method and I got 1.666666 for the x which is really close to the analytical solution. I cannot say this optimization method does not work well for this problem but I would guess it would work better than other optimization methods in other situations. It might work better in the ones with more variables or the ones with many other constraints.
I didn't get the optimization library to work for the example as I wasn't able to determine which method from the optimization library was appropriate for the example.
(Most Challenging A) Optimization Issues
General Sentiment about Optimization
(Most challenging B) Git Issues
Other Git Questions
Other Interesting Questions
ADDITIONAL QUESTIONS
Lessons about this course
0909-Optimization
Lessons about this course
Where you able to get the optimization library to work for the simple example?
No I couldn't figure out what method to use or what further information I could I have fed the minimize function to get to run. The documentation was a little hard to figure out and internet examples solved problems a bit more complex than the example we used.
I tried to solve for the local minimum with scipy optimization method and I got 1.666666 for the x which is really close to the analytical solution. I cannot say this optimization method does not work well for this problem but I would guess it would work better than other optimization methods in other situations. It might work better in the ones with more variables or the ones with many other constraints.
I didn't get the optimization library to work for the example as I wasn't able to determine which method from the optimization library was appropriate for the example.
(Most Challenging A) Optimization Issues
General Sentiment about Optimization
(Most challenging B) Git Issues
Other Git Questions
Other Interesting Questions
ADDITIONAL QUESTIONS
CMSE802
0914--AvsN-pre-class-assignment
Understanding Local Package Installs
Understanding Conda
Debugging
I cannot get Conda working. I think it's a path issue. How do I find the path where my conda is to add to my zshrc?
pip install of mdtraj, had some errors. " command errored out with exit status 1"
Is there any recommended books or videos that we can try to practice on our own? Because this part is challenging for me and I think I need more examples and practice.
More Environment questions
I would really like to see if you could run an entire project in an environment and if you could "share" that environment with another individual on another machine. Is that allowed?
Data Fitting
The example showed in this assignment doesn't have too much data. What kind of methods will we use for large dataset and how to aviod overfitting problems?
Tricks to merging Jupyter files
Tricks Merging Jupyter Files
I was trying to merge my personal branch with the origin master. While doing that, I found that the 0910 in class assignment was in conflict with the file from the origin. To resolve the conflict, I opened the file in using vi, as suggested by the tutorial video. However, although in the tutorial video, editing the README.md file seemed easy to do, I found editing the long .ipynb file to be quite cumbersome using the terminal. Is there any easier way to resolve the conflict between two .ipynb files without editing the long file in the mac terminal vi mode?
PIP and Conda Installs
Global PIP Installs
pip install PACKAGE_NAME
Requires administrative (Root) access
Hides files someplace on your computer
User PIP Installs
pip install --user PACKAGE_NAME
Hides files inside your home directory
Local PIP Installs
pip install -t ./FOLDER/ PACKAGE_NAME
Requires adding ./FOLDER/ to path every time
sys.path.append('./FOLDER/')"
Environment PIP Installs
pip install PACKAGE_NAME
Requires having pip installed in your environment.
Requires activation of the environment
Installs files in the environment folder
Base Environment
Default environment
User Environment
conda create --name ENVIRONMENT_NAME
Creates an environment at the user’s default location
Local Environment
conda create --prefix ./ENVIRONMENT_NAME
Creates an environment folder at the current location
CMSE802
0916--LSF
Questions
The most challenging part was figuring out if I plotted the "DO THIS" last parabola part right. I hope we can see the solution in class.
I think I am still confused about the Pseudoinverse part and I need more explanation about this part.
Where are we trying to go with this new info?
When I try to use git pull command on the master branch, everything went smoothly. But I noticed that the test branch didn't have any updates as I did the git pull. If I want to have these updates in my test branch as well, which command should I use?
Other Questions
Can we finish the optimization lecture from the last class?
CMSE802
0921--Stats-pre-class-assignment
6. ANOVA
Now let us assume that your advisor wants you to run an ANOVA analysis.
Where you able to get ANOVA to work?
Not quite; I found an example that uses pandas and scipy, but I could not import one of the packages that the example uses (factorial).
I did not get this working. I am so freaking confused as to what an ANOVA table is an I need to hear it from a human I can ask questions to.
I was able to get an ANOVA model working with the statsmodel library. However, I can't say that I entirely fully comprehend what the program is doing or am I able to interpret the results.
so-so; my brain soaks up these types of tools best when I use them to answer real world questions; I really should have tried to use them in this way but I just make time to do it
Using Example Code and Programming to Learn
What would you say is(are) the most important topic(s)/concept(s) covered in this assignment?
I think the most important topic was the emphasis on multiple ways to do things. You made it clear there are lots of packages and let us choose our own for ANOVA and I appreciate that freedom and information.
Questions
Other Questions
CMSE802
0923 Code Optimization
Problems logging into the HPC?
OpenOnDemand Issue
I could not clone the course repo to my OpenOnDemand account.
The terminal returned:
error: RPC failed; result=22, HTTP code = 404
fatal: The remote end hung up unexpectedly
I got a different error message when I input the wrong password, so I'm not sure if this is an authentication error or something else.
http://googleapps.msu.edu (for debugging)
Problems Cloning the course repository?
Website
https://gitlab.msu.edu/cmse802-f20/cmse802-f20
Ssh URL
git@gitlab.msu.edu:cmse802-f20/cmse802-f20.git
HTTP URL
What did you learn?
HPCC Questions
Other Questions
CMSE802
0928-Graph_Theory
Installation trouble
Were you able to find a graphing library and get it to plot the provided graph? If not, where did you get stuck. Answer: I updated my MacOS today and the terminal commands are not working for git, pip etc. I get the following error message
"xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun".
Toubleshooting
I have searched this error message on the internet and apparently I am going to need to download and install xcode. I tried to download the xcode from terminal with the following command "xcode-select --install". But this is not working and I get the following error message-
"can't install software because it is not currently available from the software update server".
Continued search
I searched this on the internet too and the suggestion is to download xcode manually from apple developer website and then install it manually. However, the xcode12 version is almost 12GB in size.
I am currently downloading the xcode12 from the developer website, it will take around 5 hours to download and then hopefully I will be able to install xcode, then install pip, and then install the necessary library package to complete the assignment. I have found a couple of graph libraries searching on the internet. I currently have my eyes on something called "networkx" which looks promising. I am currently running the notebook using the anaconda interface and not from the terminal using command line.
Ideas for solving the problem
Other Questions
CMSE802
0930 Web Scraping
Your Questions
Quick Overview of HTML
<war>Fighting</war>
Tag Attributes
<a href=”http://cmse.msu.edu/cmse802”>Course Website</a>
The “href” part is an attribute. Tags can have all kinds of attributes.
A few Common Tags
Table Example
<table style="background-color:Green">
<tr>
<th>Month</th>
<th>Savings</th>
</tr>
<tr>
<td>January</td>
<td>$100</td>
</tr>
</table>
Regular Expression Overview
Other Questions
I would like to know the difference between urllib and selenium. I believe they are both used to get webpages but you can parse through the webpage with selenium; similar to parsing through html web pages with beautiful soup.
CMSE802
1006-Graphs
How did you do on the pre-class?
I know I have worked on it before but I could only find K nearest-neighbor when searching and I couldn't figure out if this was different or the same or what was happening.
I was able to find the KNN between the random generated points but was not able to generate the graph plot.
I was able to iterate through the 50 fake towers, plot them, and find the closest tower for each. I was not able to determine how to implement the adjacency graph using a dictionary of lists and plot the connections.
I wish I had more support this time. A video explaining how the nearest neighbor algorithm would have been helpful because I had such a hard time with this part.
I was using 'plt.plot' to plot NNG, simplly loop over each points and plot a straight line to its nearest point. I am curious that can I use 'NetworkX' to plot NNG with directed edges at once.
Calculating Nearest Neighbor
Nearest Neighbor Algorithm
Loop over all of the points:
Loop over all of the other points
Calculate current_distance
If current_distance < minimum_distance
Set minimum-distance to current_distance
Set index to current_distance_index
Save index and minimum_distances for this point.
K Nearest Neighbor Algorithm
What is different?
Other Questions
In the website where towers data is downloaded form, I just saw the range in there is 3 miles rather than 4 miles.
For the nearest neighbor graph part, the graph reminds me of k-means clustering method to classify the data into several groups. So I would like to ask is there any method to run k-means with Python?
From the assignment, we were task to download for all of the towers in a 4.0 Mile radius from the MSU engineering building but the site gives 3.0 mile radius.
CMSE802
1007--PullRequest-pre-class-assignment
Upcoming Workshop On Python (Free)
UPCOMING HPC WORKSHOPS
OCTOBER 2020
PC2HPC WEBINAR: Python on HPCC
Oct 12, 10:00 a.m. -12:00 p.m.
Learn best practices to improve the performance of your Python code on HPCC.
https://icer.msu.edu/education-events/pc-hpc-python-hpcc
Questions
Automatic Documentation (Due Friday)
CMSE802
1012 - Pre-class assignment - Unit Testing
Todays Modified agenda
Pytest vs doctest
How do I decide what makes an effective test to use in unittest?"
Random Testing
Answercheck Example
git clone https://github.com/colbrydi/jupytercheck
Unwrapping Loops
Understanding the mod %
Questions
What is name==”main”?
How do you use argv?
Understanding Argv and name=”main”
argv=['first-arg-is-ignored'], exit=False
Review Tower Exersice
String Comparison
Hack-a-thon
What did you find?
What did you find - Winner Difflib
What would you say is(are) the most important topic(s)/concept(s) covered in this assignment?
Questions
CMSE802
Introduction to Classes
Class Questions
CMSE802
Agent Based Modeling
Agent Based modeling
An agent-based model (ABM) is a class of computational models for simulating the actions and interactions of autonomous agents (both individual or collective entities such as organizations or groups) with a view to assessing their effects on the system as a whole.
-- Source - Wikipedia
Questions about the code
CMSE802
Pep8 and linting
Import Question
In part 2, after running the linter, it told me that "example.py:1:0: W0611: Unused import sys (unused-import)" and "example.py:2:0: W0611: Unused import os (unused-import)". My code rating has been decreased because of these two sentences, But if I delete these two sentences, the code will be broken. What should I do about this part?
Character question?
I couldn't distinguish between the correct form and the wrong form of the Immediately before a comma, semicolon, or colon from the documentation page. They all look the same.
Another import question
Does the linting library/ program depends on the library you are using? Because i tried it on my code and I receive a message to change one of my imported libraries to another library.
CMSE802
Machine Learning
Datasets
My research is not in Machine Learning but I wish to learn more of Machine Learning.
None of these datasets relate to my own research. I am guessing that is because my research is related to the RF accelerator technology and not many experimental data is available publicly on this topic. However, the generated datasets section seem to be interesting. Maybe I could try to generate something useful playing with that tool.
Uploading Documents to the HPCC
I would like to know if we could go over how to ssh upload a document to HPC because I cannot remember and didn’t get anything I understood fully on Google. I would rather ask you than break something on accident by using code I didn’t understand on HPC.
scp sourcepath destinationpath
path USERNAME@URL:~/LOCALPATH
Quick check-in
CMSE802
Introduction to Neural Networks
UNIVERSAL APPROXIMATION THEOREM
Great cosmic power…. Itty bitty living space.
NN Questions
Exploring a network
Programming Questions
CMSE802
Back Propagation
Final Project Due Dates
Clarifications from last time
Why is the activation function sigmoid?
Neural Networks can not model any function. They can approximate any function. Slight but important difference.
Learning, m and b from y=mx+b turns out to be much thicker than I thought. It is a good exercise if you want to try it.
Neural Networks are Great
Neural Networks have issues
* To be fair. This is true with almost all machine learning.
Some current research in Neural Networks
Note, these are your instructor’s terms. I am not fully versed in the current NN terminology so ideas may vary...
Other Questions?
CMSE802
Tensorflow on the HPCC
Installing Anaconda and Tensorflow
Setting up Conda Environments
The Google Colab
Where you able to get the example working?
Challenging Parts
The Downsides to using Colab?
User Interfaces
Graphical User Interfaces (GUI)
Command Line Interfaces (CLI)
Advanced Programming Interfaces (API)
Easier for software users
Easier for programmers
How often does this code be used?
Who is your user?
Goal: Make it as easy as possible to automate
Whenever possible interface design should be done in layers:
If done correctly, each level gets easier to program. We can also make the code modular and therefore more flexible.
Modular Interface Design
Data interface
Read/Write Data
Convert Data
Check for errors
Etc.
Computation interface
Run statistics
Model Simulation
Calculate Results
Etc.
User interface
Generating Graphs / Charts
Asking for feedback
Etc.
Ordinary Differential Equations
Ordinary Differential Equations
it was a great start but i'm not entirely sure how to determine when an ODE is the best way to model a phenomenon
How can we know which equation are we needed for our project? How to build up ODE for our project?
Windows vs Linux
File Paths
Questions
I cannot set-up the correct directory for "!python searchnb.py -s "beautiful" --directory "~/"" . the input "~/" did not work in my windows laptop.
File Paths
Windows
C:\\Documents\ and\ Settings\\User\\sdcard.img
C:/Documents\ and\ Settings/User/sdcard.img
Mac / LInux
/mnt/home/my\ Documents/
from pathlib import Path
The pathlib library is designed to help fix problems with paths inside of python.
We should get into the habit of always using it with relative paths. For example
from pathlib import Path
mypath = Path(‘./thisfile’)
with open(str(mypath) as file:
contents = file.readlines()
This library seems to prefer the linux/mac path syntax (it is considered more standard).
File Formats
Encoding problems
In "4. Practice", the code was not working and returned error as follows:
Traceback (most recent call last):
File "searchnb.py", line 22, in <module>
for line in fp:
File "D:\Software\Python3.7\setup\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1070: character maps to <undefined>
To fix that, I add "encoding="utf8"" to line 21 so that it became:
"with open(filename,'r', encoding="utf8") as fp:"
This modification did fix the error even though I do understand the logic behaind.
Not all “Text” is created equal
ASCII
UTF8
Many others
Scientific Image Understanding
Questions
I used the Jupyterhub and failed to install environment for see-segment project with the error "[Errno 122] Disk quota exceeded".
I am having problem with cloning repositories by getting an error message that. So I was not able to do that.
Yes, however the conda environment created and activated didn't have 2 packages despite the terminal suggesting they'd been installed properly. Once they were installed the example file worked well.
Code Review
Code Review Comments
The checklist was helpful but i wonder whether i'm qualified to evaluate all of the things i should.
It feels like we’ve already done this in practice during the semester. So this is kind of redundant.
If I haven't finished my code yet, do I still have a change to do the code review?
Today’s Agenda
setup.py
Sharing your package with the world!
Problems with setup.py
What is a package?
Find a url to generate a python package
Github.io